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ABSTRACT 

c- •« i " t/ 5 > 

A study was conducted to rdei'emine' the relative strengths and 
basic rationale for government use o’fWarious institutions to conduct 
aeronautical research^, and development a(R&D) for the Federal Government. 
The nation's aeronautical :R© »ihstitutibns?wer.e;.diy4dqd ‘^hto five 
categories: manufacturing companies ,’ service R&Q ^comp,bgi'es, non-profit 
R&D institutions, universities, and government laboratories. The 
relative roles of these institutions. were .delineated in regard to con- 
ducting basic research, 'applied research, technology advancement, and 
development in aeronautics for the 'Government 1 : -The - relative strengths 
and roles of the National Aeronautics and Space Agency (NASA) labora- 
tories were evaluated within the total context of all of the nation's 
aeronautical R&D institutions. 

A total of 25 of the nation's leaders in aeronautical R&D 
participated in the study. There was an equal number of leaders from 
each of the five main aeronautical R&D institutions in the United 
States. The investigation was conducted using both a survey question- 
naire and a new conferencing method called the tributary technique. 

The tributary technique was devised to obtain direct quantitative 
measurements of subjective judgments by groups that have both common 
and highly competitive objectives. 

The nation's aeronautical R&D institutions have characteristic 
differences which make them individually suitable for particular types 
of work. The basic strengths of these institutions produce a pattern 
of complementary capabilities that properly overlap, but the nation is 
fortunate in having at least one or two institutions that are partic- 
ularly well suited for each category of work. For example, it was 
found that government laboratories, in general, and especially NASA 
laboratories, should be the prime national producers of applied 
research in aeronautics. American aeronautics needs the new stimulus 
of markedly increased outputs of broad-based innovative. research from 
NASA laboratories more than it needs most of the technology advance- 
ment and development-oriented programs currently underway in these 
laboratories. The Government should use manufacturing companies for 
the vast bulk of development, most technology advancement, and a 
substantial amount of applied research in aeronautics. However, the 
Government will have to implement programs to encourage the transfer 
of full information on technology and research advancements, from the 
companies that do this work for the Government, to competing companies 
that also need the results. Universities should be the primary sources 
of basic research. Service R&D companies and non-profit R&D institu- 
tions provide valuable, specialized, supplementary technical capa- 
bilities and other unique attributes, which together span the entire 
spectrum of aeronautical R&D. 
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PREFACE 

. _ -.a 

This investigation was made possible through the cooperation 
of 27 separate organizations in Government and the private sector. 

The study presents subjective judgments on fundamental rationale as to 
why the Federal Government should use particular segments of the 
aeronautical community to conduct basic research, applied research, 
technology advancement, and development in aeronautics. A total of 25 
of the nation's leaders in aeronautical research and development (R&D) 
participated in the study. The participants are listed in the next 
section. All participants were directly responsible for work con- 
ducted for the Federal Government. Thus, tney all had competitive 
vested interests in the results of this study. Many of them had 
divergent personal views on the subject as a result of years of experi- 
ence within different segments of the aeronautical R&D community. 
Therefore, this investigation had substantial potential for producing 
conflicting viewpoints among the participants. 

Similarly, the reader will undoubtedly have his own initial 
views on these issues, if he has been concerned with aeronautical R&D. 
This publication is arranged in a form that will allow the reader to 
review the processes, determinations, and logic that the participants 
went through during the course of the investigation. Even the author 
was surprised at the extent of agreement that emerged within and among 
the various groups of participants. The data and observations indi- 
cated that the participants were profoundly influenced in their 
thinking as they came to grips with basic issues involved in deter- 
mining where particular segments of the aeronautical R&D community best 
fit into the overall structure of work for the Federal Government. 
Hopefully, the reader will also gain an expanded understanding of the 
relative strengths and importance of the various segments of the 
aeronautical R&D community as he reviews judgments presented from dif- 
ferent vantage points. 
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interpretation, and presentation of the results. The author also 
acknowledges the contribution of Stephen H. Achte.'hagen who periodi- 
cally reviewed the progress of the investigation, influenced the 
methodology, and provided many fine ideas on handling the data. 
Additionally, thanks are due to Wayne L. Strom who, during his teach- 
ings on behavioral theory, laid the groundwork from which the author 
was able to develop the tributary conference technique described in 
this publication. Kenton L. Anderson, in his teachings; provided sub- 
stantial insight into the characteristics of organizations, the factors 
that differentiate them, and the elements of effective organizational 
structures. Lisa McCormack and various members of the Dean's office of 
the School of Business and Management provided valuable administrative 
assistance for this project. Ruth D. Atteberry, Associate Dean, 
provided excellent editorial review and advice on preparation of this 
publication. 

This investigation partially was sponsored by the National 
Aeronautics and Space Administration, NASA-Ames Research Center, under 
Grant NSG-2159 which began on June 1, 1976. The author is indebted to 
Harry Hornby, Grant Technical Monitor, and J. Lloyd Jones, Chief, Plans 
and Analysis Office, NASA-Ames Research Center. They were responsible 
not only for initiating and monitoring this study, but they also 
provided some excellent ideas and substantial encouragement during the 
course of the investigation. Finally, thanks are due to Hans Mark, 
former Director of NASA-Ames Research Center, from whom the author has 
learned a great deal about R&D organizational management over the past 
three years. 
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Chapter 1 _ . . ; ^ . 

SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 

This chapter presents an executive suimiary, conclusions, and 
recommendations from the investigation discussed in this publication. 

GENERAL 

This investigation was conducted to determine what relation- 
ships, if any, exist among the relative strengths of the nation's 
aeronautical research and development ( R&D) institutions for con- 
ducting basic research, applied research, technology advancement, and 
development in aeronautics for the Federal Government. A total of 25 
of the nation's leaders in aeronautical R&D participated in the study. 
There was an equal number of leaders from each of the five main 
aeronautical R&D institutions in the United States: manufacturing 

companies, service R&D companies, non-profit R&D institutions, 
universities, and government laboratories. The institutions and areas 
of work are defined in Appendix B on pages 174 through 176. The 
investigation was conducted using both a survey questionnaire and a new 
conferencing method called the tributary technique. The tributary 
technique was devised to obtain direct quantitative measurements of 
subjective judgments by groups that have both common and highly 
competitive objectives. The participants were able to express their 
judgments in confidence since they could be identified only by 
institutional affiliation. The participants' points of view were cast 
in the form of consensuses of judgments for the total group of all 
participants and for the sub-groups from the various institutions. The 
results indicated that there was significant agreement among the total 
group of all participants on 96 percent of their judgments. The level 
of agreement within any group was taken as being significant only when 
there was a probability of at least 95 percent that the group con- 
sensus judgment was the result of a meaningful group determination 
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rather than chance . The probabi 1 i tv of^ chance for many .judgments wa s 
less than 1 in 1000. 

The consensus judgments by the sub-groups representing the dif- 
ferent aeronautical R&D institutions were often quite different from 
each other. Natural self-oriented biases frequently influenced the 
judgments of the sub-groups from the various institutions. However, 
a careful analysis of all judgments in the study indicates that insti- 
tutional biases essentially were balanced out by using consensus 
judgments based on averages of the numerical judgments of all 25 
participants. These consensus judgments gave equal weights to view- 
points from the various institutions. The conclusions presented in 
this section were therefore based upon the consensus of judgments of 
all the participants as a group. 

There are too many conclusions that can be drawn from the 
wide range of results obtained from this investigation to do justice to 
them all. The divergent views of the sub-groups and the nature of 
biases may be very important relative to many issues. It also may be 
vital to know which rationales were rejected and what the extent of . 
agreement was on these viewpoints. In fact, the reader may often wish 
to draw his own conclusions from the results relative to specific 
issues at hand. 

However, there are some important general conclusions and 
recommendations that can be formulated, based on the overall views of 
the entire group of aeronautical leaders. First, one must accept the 
premise that this total group of aeronautical leaders was probably as 
knowledgeable about America's aeronautical R&D institutions as any 
group that might be convened. It then stands to reason that signifi- 
cant agreement among these aeronautical leaders, as a group, probably 
yielded correct determinations of the relative strengths, supporting 
rationale, and best roles for these institutions in regard to con- 
ducting aeronautical R&D for the Federal Government. 

The following general conclusions are based upon the overall 
group consensus judgments of the 25 aeronautical leaders and other, 
information gained during this investigation. General conclusions are 
presented for each of the five aeronautical R&D, institutions, in turn. 



and then for the laboratories of the National Aeronautics and Space 
Administration (NASA), which are a special case within the general ' 
category of government laboratories. 

MANUFACTURING COMPANIES 

Manufacturing companies are. strongly oriented, in all their 
R&D activities, toward achieving competitive, profitable, manufactured 
products. Their highly directed attention to end products yields 
characteristics that coincide, better than those of any other aero- 
nautical R&D institution, with the Government's need for highly effec- 
tive, concentrated, arid well organized efforts in the final stages of 
the R&D process. At the same time, the product orientation of manu- 
facturing companies greatly reduces their usefulness to the Government 
for work in the early stages of aeronautical R&D and limits their 
technical activities in other ways that broadly impact the progress of 
American aeronautics. 

Development . Manufacturing companies are by far the most 
logical institutions to conduct the vast bulk of development work in 
aeronautics for the Federal Government. Manufacturing companies have 
the greatest ability to produce a totally satisfactory product, 
including post-manufacturing support. -The potential for volume pro- 
duction provides manufacturing companies the proper motivation to carry 
development programs the entire way from inception to successful 
manufactured products. Manufacturing companies characteristically 
produce the highest quality of development work. They have the highest 
ability of any of the aeronautical R&D institutions to focus on 
achieving solutions to specific real problems. Government funding for 
development programs in manufacturing companies helps to maintain 
their world-wide competitive positions. Such funding is essential in 
order to create new products that are in the national interest, when 
these development programs are beyond the financial capability of 
industry. 

However, manufacturing companies should not have more than a 
very small percentage of the total capital value of major aeronautical 
experimental facilities built by the Government. Manufacturing 
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companies are also not the best sources for impartial technical guid- 
ance to the Government on development contracts to other organizations. 

Technology advancement . Manufacturing companies should play 
the dominant role in aeronautical technology advancement for the . 

Federal Government. They have the facilities (manufacturing) and the .... 
knowledge (technology) to do it. Manufacturing companies character- 
istically produce the highest quality of technology advancements and 
they have the greatest ability to focus their efforts on achieving 
important gains. The Government should fund technology advancement / 
work in manufacturing companies so that they can stay abreast of new 
advancements and gain a world-wide competitive advantage. Furthermore* 
technology advancement is part of the logical process of developing 
needed high technology products. It is also important for manufac- 
turing companies to conduct technology advancement for the Government 
in order to define cost performance, risks, and profit potential of 
possible new products. ■ 

However, there is a serious drawback in placing the vast bulk 
of the Government's contracts for technology advancement with manu- 
facturing companies. Manufacturing companies are characteristically 
the least willing of any aeronautical R&D institution to provide outside 
organizations with full information on the results of these programs. 
Nevertheless, the need to use manufacturing companies to advance - 
aeronautics technology is too compelling to direct most of this work 
into more open aeronautical R&D institutions. This dilemma cannot be 
reconciled unless the Government's intervention in the normal process 
of industry competition, by funding technology advancement in certain 
firms, is also accompanied by effective governmental measures to help 
all aeronautical firms share fully in the results. . j 

Manufacturing companies have next-to-the-lowest ability of all 
five aeronautical R&D institutions to provide the Government with 
impartial technical guidance on technology advancement contracts; with 
other organizations. The Government needs other institutions for this 
function. 

Applied research . Manufacturing companies should play the 
second-greatest national role, among the five institutions, in pro- 
viding applied research in aeronautics for the Government. Applied 
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research is part of the logical process of developing needed high 
technology products in manufacturing companies, Applied research will 
help manufacturing companies stay ahead in development and gain a 
national competitive advantage. Manufacturing companies sometimes have 
unique capabilities to carry out applied research by virtue of their 
proximity to development problems. When applied research is conducted 
in manufacturing companies it helps them attract, develop, and retain 
people with needed skills. 

There are some significant factors that should moderate the 
extent of the role played by manufacturing companies in conducting 
applied research for the Federal Government. Manufacturing companies 
characteristically produce nearly-the-lowest quality of applied 
research among the five aeronautical R&D institutions. Manufacturing 
companies are the least willing of all aeronautical R&D institutions to 
provide full information on the results of their government-funded 
research. While applied research in manufacturing companies helps them 
guide their own organizations' work, it does not help the Government 
guide its contract work with other organizations.. Manufacturing 
companies are the least able of any institution to provide the 
Government with impartial technical guidance on applied research con- 
tracts with other organizations. In addition, research groups in 
manufacturing companies will not have constant access to major 
government-built aeronautical experimental facilities, which might be 
advantageous to research. Only a small percentage of these facilities 
should be located at or under the control of manufacturing companies. 

Sasic research . Manufacturing companies should play the 
smallest role of any aeronautical R&D institution in government-spon- 
sored basic research. However, they should do some basic research. 
Sometimes a manufacturing company has a unique capability to conduct- 
basic research of a particular type. Basic research provides manu- 
facturing companies with new advancements. Working in 

the field helps manufacturing companies capitalize on these new 
findings by others and it allows them to move these advancements into 
the applied research phase. The work in basic research gives manu- 
facturing companies a far-term perspective. It helps them avoid 
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near-term focusing of their R&D efforts and thus prevents technological 
surprise. 

However, manufacturing companies characteristically produce the 
lowest quality of basic research of the five institutions. They are 
the least willing to provide full information on their results. Manu- 
facturing companies should not have more than a smal 1 percentage of the 
major, government-built, experimental facilities that might also ; be 
useful in basic research. While basic research will help manufacturing 
companies in guiding their other work, it will not materially aid the 
Government in obtaining technical guidance for its other basic research 
contracts. Compared to other aeronautical R&D institutions, manu- 
facturing companies are characteristically the worst sources of impar- 
tial technical guidance to the Government on basic research contracts 
with other organizations. 

SERVICE R&D COMPANIES 

Service R&D companies tend to be highly effective technical 
organizations in certain specialty areas. They are oriented to pro- 
viding services that augment permanent staffs in government labora- 
tories, industry, and non-profit R&D institutions. The Government can 
draw on the available pools of specialized talent in service R&D 
companies without creating as permanent an investment in personnel as 
would be required by conducting such work in government laboratories. 
While the specialized roles of service R&D companies should span the 
entire spectrum of aeronautical R&D, their greatest natural roles 
relative to other institutions tend to be in technology advancement and 
applied research. Service R&D companies have a characteristic 
capability to focus on achieving solutions to specific real problems 
that is second only to that of manufacturing companies. Service R&D 
companies tend to be more willing to supply full information on the 
results of government-funded work than manufacturing companies, but 
service R&D companies are not so open in this respect as are the other 
aeronautical R&D institutions. Service R&D companies should have a 
minor proportion of the nation's major aeronautical experimental 
facilities built by the Federal Government. 
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Development . Service R&D companies should play a small role in 
conducting aeronautical development work for the Government. The 
activities of service R&D companies in conducting development should be 
far below that of manufacturing companies, but roughly on a par with 
government laboratories. Service R&D companies may achieve the 
most-effective product development in technical areas where they have 
unique capabilities. This may well be the case for certain relatively 
small aeronautical products. Service R&D companies may provide an 
economical source of development. In general, the quality of develop- 
ment work in service R&D companies is second only to that of manufac- 
turing companies. Service R&D companies characteristically h$ve at 
least as good an ability to provide impartial technical guidance to the 
Government on development contracts as other institutions in the 
private sector. 

Technology advancement . Service R&D companies should play a 
small but significant overall role in technology advancement for the 
Government. Their role in advancing aeronautical technology should be 
second only to that of manufacturing companies, but about equal to the 
role of government laboratories. Service R&D companies have special 
capabilities and economically can supply technology advancements in a 
timely fashion. Service R&D companies are able to tie technology to a 
product or other end application. In general, the quality of work in 
technology advancement produced by service R&D companies is second only 
to that of manufacturing companies. However, service R&D companies are 
not the best sources for impartial technical guidance to the Government 
on technology advancement contracts with other organizations. 

Applied research / Service R&D companies should play a small 
but significant national role in conducting applied research in aero- 
nautics for the Federal Government. Service R&D companies supply 
applied research economically and in a timely fashion. The quality of 
their applied research is characteristically about average among the 
five aeronautical R&D institutions. However, service R&D companies 
rank next to last in their ability to provide the Government with 
impartial technicalguidance on applied research contracts with other 
organizations. 
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Basic research . Service R&D companies should probably play 
only a very minor role in basic research. The extent of their activi- 
ties in basic research should be next to the lowest among all aero- 
nautical R&D institutions. The quality of basic research in service 
.R&D companies tends to be lower than that of all the aeronautical R&D 
institutions, except manufacturing companies. The ability of service 
i R&D companies to provide the Government with impartial technical guid- 
ance on basic research similarly tends to be next to the lowest of the 
five aeronautical R&D institutions. 

NON-PROFIT R&D INSTITUTIONS 

Non-profit R&D institutions are similar to service R&D compa- 
nies in that they supply valuable supplementary capabilities in aero- 
nautical R&D for the Government. Non-profit R&D institutions sometimes 
have unique technical capabilities in special fields. The role of 
non-profit R&D institutions in performing aeronautical R&D spans the 
entire spectrum of R&D. However, as producers of aeronautical R&D, 
non-profit R&D institutions play their greatest roles in basic and 
applied research. Non-profit R&D institutions should have a small 
percentage of the major aeronautical experimental facilities built by 
the Government. 

The organizational features of non-profit R&D institutions also 
make them uniquely useful to the Government for certain other aero- 
nautical R&D tasks. They can be used by the Government for objective 
technical. analyses, technical support, and for second sources on 
technical evaluations. Non-profit R&D institutions do not compete with 
industry by manufacturing products nor do they have stockholders 
demanding profits, dividends, and capital gains. Non-profit R&D insti- 
tutions can thus work with 'ndustry and can protect proprietary infor- 
mation in a fashion similar to government organizations. At the same 
time, non-profit R&D institutions are in the private sector and they 
are not subject to many of the constraints placed on government 
organizations. Thus, government organizations can use non-profit R&D 
institutions to add technical competence and flexibility to their 
capability to accomplish the Government's portion of the technical work. 
Non-profit R&D institutions tend to have a greater ability to focus on 
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solving specific real problems than do government laboratories. Non- 
profit R&D institutions are exceeded in this respect only by manufac- 
turing companies, and service R&D companies. However, non-profit R&D 
institutions are more willing to provide outside organizations full 
information on the results of their work for the Government than are 
manufacturing companies or service R&D companies. 

Development . Non-profit R&D institutions should' bonduct only a 
very small amount of the total development work in aeronautics for the 
Government. In fact, non-profit R&D institutions should be the 
smallest participant in aeronautical development of any R&D institu- 
tion, except universities. Non-profit R&D institutions may have cer- 
tain unique capabilities in specialized areas which can be of value in 
performing aeronautical development work. However, the quality of the 
aeronautical development work by non-profit R&D institutions tends to 
be next to the lowest among the five aeronautical R&D institutions. 

On the other hand, the ability of non-profit R&D institutions to pro- 
vide the Government with impartial technical guidance on development 
contracts is characteristically at least as good as any other R&D 
institution in the private sector can provide. 

Technology advancement . The relative role of non-profit R&D 
institutions in aeronautical technology advancement for the Government 
should be small, but greater .than their role in development. Non- 
profit R&D institutions should play next-to-the-smallest role in 
performing technology advancement activities in aeronautics for the 
Government. The Government can use non-profit R&D institutions on the 
government side of its aeronautical R&D activities and yet circumvent 
federal restraints. Sometimes there are special government problems or 
needs where it is necessary to have a non-profit R&D institution con- 
duct technology advancement in order to utilize their non-product 
objectivity, that is, a lack of commercial bias and absence of coupling 
to manufacturing. The ability of non-profit R&D institutions to pro- 
vide the Government technical guidance on technology advancement 
contracts in aeronautics is second only to that of government labora- 
tories. Non-profit R&D institutions provide a less permanent commit- 
ment to personnel than government laboratories. The characteristic 
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j quality of technology advancement by non-profit R&D institutions is 

average among the five aeronautical R&D institutions and it is roughly 
i on a par with the quality of technology advancement work by government 

laboratories. . • 

Applied’ research . Non-profit R&D institutions should share 
[ significantly in conducting applied research in aeronautics for the 

j . : Federal Government. The total extent of their applied research r activi- 

i : ties should add up to substantially less than that of government 

I laboratories and manufacturing companies, but it should be about equal 

j to the roles played by universities and service R&D companies. Non- 

profit R&D institutions have special organizational characteristics of 
use to the Government, as mentioned in the previous section. Non- 
| profit R&D institutions bridge the gap between universities and govern- 

ment organizations. The quality of aeronautical applied research from 
non-profit R&D institutions tends to be second only to that of govern- 
ment laboratories. Non-profit R&D institutions should be used by the 
j Government for applied research if they have the physical resources to 

do the work required. Their ability to provide the Government with 
impartial technical guidance on applied research contracts is second 
only to the ability of government laboratories. 

Basic research . Non-profit R&D institutions should play a 
relatively small, but significant, role in aeronautical basic research 
j for the Government. The total extent of their activities in basic 

: i • . : 

research should amount to substantially less than those of universi- 
! ties and government laboratories. However, non-profit R&D institutions 

: should probably play a larger role in basic research for the Government 

I 

than service R&D companies or manufacturing companies. The quality of 
J. basic research in non-profit R6D institutions is characteristically 

second only to that of universities. The organizational form of non- 
profit R&D institutions allows them to work for the Government in a 
fashion similar to government laboratories and yet circumvent federal 
constraints. They can function as captive laboratories to the Govern- 
ment and perform basic research on a long-term basis. Non-profit R&D 
institutions should be used for basic research if they have the 
necessary talent and the physical resources. However, non-profit R&D 




institutions have only an average ability, relative to the other | - 

institutions, to provide the Government with impartial technical advice 
on basic research contracts with other organizations. - 

UNIVERSITIES J . , 

The very characteristics of universities make them ideally f 


suited to be the leading institution for conducting basic research in r j 

aeronautics for the Federal Government. Their orientation towards 
education, which is the primary purpose of their institution, creates 
a climate for concentrating on the fundamentals of aeronautics. Their 
occupation with students, inquiry, and dealing with basics completely 
pervades the institution's activities. The general orientation of 
universities attracts people suited to learning and teaching by virtue ■ 

of intelligence and independent nature. Universities are naturally ! 

; 

the most willing, of all aeronautical R&D institutions, to provide \ 

full information on the results of their work for the Government. ! 

There is a strong element of striving for objective inquiry that has 
also been deemed useful in providing technical aovice to the 
Government. 

However, the same strong orientations that are so valuable for 
basic research in universities make them considerably less useful for 
more applied tasks. Universities tend to have the lowest ability to 
focus on achieving solutions to specific real problems relative to j 

other aeronautical R&D institutions. Thus, universities are primarily | 

research institutions, with the greatest emphasis on basic research. j 

Universities should have a small percentage of the major aeronautical j 

experimental facilities built by the Government. 

Development . The universities have essentially no role in 
aeronautical development. for the Government. No significant rationale ; 

has been found to support university activity in development for the 
Government. Universities characteristically produce aeronautical- 
development work of lower quality than that of any other aeronautical 
R&D institution. Universities have the lowest ability to provide- tech- 
nical guidance to the Government on aeronautical development contracts. 

In spite of their probable impartial ity, it does not seem wise to use 



disproportionately large numbers of university professors on govern-, 
ment advisory groups for aeronautical development programs. 

Technology advancement . Universities should play only a minute 
role in aeronautical technology advancement for the Government.- The 
universities' activities in technology advancement should.be substan- 
tially, less than, the roles played by all other aeronautical R&D ; - 
institutions. However, there are some instances where universities can 
provide unique sources for technology advancement. A small amount of 
work in this area can sometimes further the training of students for 
the later practice of engineering. It would also help universities in 
interfacing better with industry. The presence in universities of 
fundamental scientists contributes to the innovative thinking process 
essential to the achievement of large technology jumps. On the other 
hand, universities characteristically produce the lowest qual ity of 
work in technology advancement of any aeronautical R&D institution. 

They have the lowest ability to provide the Government with technical, 
guidance on technology advancement contracts with other organizations, 
even though university advisors are likely to be impartial . As in the 
case of development, it does not seem appropriate to use dispropor- 
tionately large numbers of university professors on senior government 
advisory groups that influence critical decisions on technology 
; advancement programs in aeronautics. 

Applied research . Universities should have a substantially 
lesser role in conducting applied research in aeronautics for the 
Government than the roles of either government laboratories or manu- 
facturing companies. However,, the extent of university activities in 
applied research should be roughly on a par with service R&D companies 
and non-profit R&D institutions. Applied research is an integral part 
of the educational process in engineering schools. Graduate engineer- 
ing students, who are low cost, energetic, and questioning, are often 
strongly motivated towards applied research. Universities sometimes 
have the unique facilities and the talent to do applied research. The 
long lifetimes typical of some applied research projects are com- 
patible with university patterns. On the other hand, the quality of 
university-applied research in aeronautics is characteristically below 
that of almost any other aeronautical R&D institution. Universities 
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are about average, among the R&D institutions, in their ability to pro- 
vide the Government impartial technical guidance on applied research 
contracts with other organizations. 

Basic research .. Universities should perform the major national 
role in providing aeronautical basic research for the Government. In 
fact, the role of universities in providing aeronautical basic research 
for the Government should be roughly equivalent to the combined roles 
of all other aeronautical R&D institutions. Universities have atmo- 
spheres that draw people qualified by independence and motivation to do 
basic research. Graduate students, who are low cost, energetic, and 
questioning, have an idealistic approach to basic research. Universi- 
ties characteristically perform the highest quality of basic research 
of any aeronautical R&D institution. Basic research is an integral 
part of the education process. Universities should be funded by the 
Government to do basic research to further the development of specific 
capabilities that are in the national interests. Universities are 
probably second only to government laboratories in ability to provide 
the Government with impartial technical guidance on basic research 
contracts. 

GOVERNMENT LABORATORIES 

The government laboratories provide important technical contri- 
butions to aeronautical R&D and play vital central management roles 
within the total complex of America's aeronautical R&D institutions. 
Government laboratories gain their natural orientations and their most 
important capabilities from three main factors: (a) their positions 

within the Government, (b) the knowledge they derive from the technical 
work they perform, and (c) the substantial resources they can direct 
toward technical objectives. The most important role of government 
laboratories is in technical guidance within the Government for plan- 
ning and executing aeronautical R&D programs. Government laboratories 
characteristically have the greatest ability, of any aeronautical- R&D 
institution, to provide impartial technical guidance to the Government 
on contracts from basic research through development. 
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Government laboratories are vital to communications and working 
relationships between the Government and the nation's other aeronauti- 
cal R&D institutions. Government laboratories should be advocates, 
critics, and interpreters within the Government, of advancements that 
have been or could be made by all of the nation's aeronautical R&D 
institutions. Government laboratories should play a major role in 
translating the nation's aeronautical ■ needs into specific research and 
technology advancement tasks. They have the resources to move funda- 
mental advancements into the more applied stages of the R&D process. 
Government laboratories should strive to increase the overall national 
capability in aeronautical R&D. Government laboratories should not 
only produce technical advancements, in-house and under contract, but 
they should also use all their influence and resources to insure that 
the results of R&D purchased with tax dollars are made widely available 
to all American aeronautical R&D institutions. Government laboratories 
are second only to universities in their willingness to provide outside 
organizations with full information on government- funded research and 
technology. 

Government laboratories should conduct some in-house technical 
work in all phases of aeronautical R&D in order to provide skills, 
knowledge, and understanding so that they can properly perform their 
other broad government roles in aeronautical R&D. They have the most 
diversified natural capabilities of any aeronautical R&D institution, 
as reflected by their being the only institution which produces a 
quality of work that characteristically ranks in the upper half of 
institutions over the entire aeronautical R&D spectrum. However, 
government laboratories are best suited and are most needed as major 
producers of basic and applied research in aeronautics. The position 
of government laboratories within the Government also leads to a lack 
of orientation and creates physical restrictions that prevent them from 
being top performers in technology advancement and development work. 
Government laboratories' characteristically have next- to- the- lowest 
ability to focus on achieving solutions to specific real problems com- 
pared to the other aeronautical R&D institutions. Government labora- 
tories should be the guardians of most of these government facilities 
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so that the results from these facilities and access to them will be 
available to all segments of the aeronautical R&O community. 

The role of government laboratories as a primary producer of 
research is greatly enhanced by their access to. major aeronautical 
experimental facilities built by the Government. However, a large 
portion of major government experimental facilities; should also be 
made easily available to other aeronautical R&D institutions by placing 
them in central government test centers which do not have their own 
resident government R&D groups. 

Development . Government laboratories should play only a very 
small role in conducting their own development work. They should do 
development work in areas where they already have a unique capability 
for it. Development work by government laboratories provides an 
arsenal system of manufacturing in areas where there is little 
industrial/coiranercial interest. Government laboratories must be 
engaged in development to provide knowledge and life cycle laboratory 
support for operational systems. However, government laboratories 
characteristically produce about average quality of development work 
relative to the other four aeronautical R&D institutions. Thus, 
government laboratories should not be major development organizations. 

Technology advancement . Government laboratories should play a 
small but significant role in providing aeronautical technology 
advancements. Their in-house efforts in technology advancement should 
be far below that of manufacturing companies. : Government laboratories 
and service R&D companies should be the two greatest sources of tech- 
nology advancements in aeronautics. Technology advancement should be 
done within government laboratories in order to provide knowledgeable 
buyers of the developpd product. Technology funded within; government 
laboratories often provides unique facilities accessible to development 
groups in the public and private sectors. Government laboratories have 
resources to support prototype development, which are notiavailable tc 
a single industrial organization. Technology that is funded within the 
Government becomes available to all aeronautical R&D institutions, 
which helps to maintain an open competitive environment for the 
developed product. However, government laboratories characteristically 
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produce only average quality of work in technology advancement relative 
to the other four aeronautical R&D institutions. 

Applied research . Government laboratories should be the 
greatest source of applied research in aeronautics for the Government. 
Government. laboratories characteristically produce the highest quality 
of applied research of any aeronautical R&D institution. Government 
laboratories should conduct applied research to provide knowledgeable 
leadership for coordinating and guiding aeronautical R&D. Applied 
research should be done by government laboratories to provide quick 
response capabilities to solve the Government's operational systems 
problems. Large-scale facilities which might be required for applied 
research can be best built and operated by the government laboratories. 
Government laboratories should conduct applied research when there is a 
lack of conmercial incentives. The applied research done by government 
laboratories provides them information, skill, and knowledge necessary 
to permit them to conduct independent evaluations of competitive R&D. 

Basic research . The government laboratories should be second 
only to universities, among all aeronautical R&D institutions, as the 
nation's primary source of government- funded basic research in aeronau- 
tics. Basic research conducted within government laboratories facili- 
tates a transfer of basic research knowledge into applied research. 
Basic research in government laboratories provides a reservoir of 
aeronautical knowledge in the public domain. Government laboratories 
should particularly conduct basic research in cases where the research 
can benefit from any unique facilities they may have. Government 
laboratories could provide a stable environment for long-term,, 
high-risk research. 

NASA LABORATORIES 

The current conclusions regarding the relative strengths and 
roles of NASA laboratories are based on consideration of the roles 
played by all aeronautical R&D institutions. The laboratories in NASA 
fall within the general class of government laboratories. However, 
there are substantial differences between NASA laboratories and other 
government aeronautical laboratories. Three government agencies have 
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their own laboratories engaged in aeronautical R&D: the Department of 

Defense (Army, Navy, and Air Force), the Department of Transportation, 
and the National Aeronautics and Space Administration. At present, 

NASA is the only one of these three agencies that is not itself a 
prime customer for the final products resulting from its in-house and 
contracted aeronautical, R&D. NASA is the only agency that does not 
conduct regular aeronautical operations using these products. NASA is 
not a major developer nor a purchaser of civil aircraft. These activi- 
ties are the respective domains of aircraft manufacturing companies and 
commercial airlines. Furthermore, the National Aeronautics and Space 
Act of 1958 does not give NASA the roles of coordinating national aero- 
nautical R&D nor providing advisory services to other aeronautical 
institutions as was legally required of its predecessor, the National 
Advisory Committee for Aeronautics (NACA). The NASA is an independent 
government agency charged with promoting advances in aeronautics for 
other agencies and institutions. Therefore, the general conclusions of 
this study for government laboratories musr be made more specific for 
NASA laboratories. 

The laboratories in NASA should be prime producers of basic and 
applied research in aeronautics. American aeronautics needs the new 
stimulus of massive outputs of broad-based innovative research from 
NASA laboratories more than it needs most of the specialized technology 
advancement and development-oriented programs currently underway in 
NASA laboratories. NASA laboratories are in excellent central posi- 
tions to conduct research by virtue of their locations in a separate 
government agency. NASA laboratories have a natural orientation for 
broad-based, systematic, and highly innovative research. The research 
in NASA laboratories will aid NASA in guiding its own aeronautical P&D 
contract programs. It will also assist them in certain high-level 
government advisory functions and in long-term studies of the national 
benefits to be gained from aeronautical R&D. 

The rationale that indicates that government laboratories, in 
general, should be prime producers of aeronautical basic and applied 
research largely holds true for the specific case of NASA laboratories. 
In fact, much of this rationale stemmed from consideration of NASA 
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laboratories. The NASA laboratories have many of the same character- 
istics attributed to the government laboratories, such as their 
exceptionally great willingness to provide full information on the 
results of government-sponsored R&D. The NASA laboratories should 
share in the role of other government laboratories as users and guard- 
ians of a major portion of the nation's large-scale aeronautical 
experimental facilities built by the Government. These major NASA 
• experimental facilities make it possible for NASA laboratories, in 
particular, to be mass producers of high-quality innovative research. 

Development . Perhaps a quarter of the current work by NASA 
laboratories in aeronautics is so highly oriented toward specific 
applications and hardware that it must be considered in the development 
phase of aeronautical R&D. NASA laboratories should be hardly involved 
in in-house development activities in aeronautics. There is no valid 
rationale for NASA laboratories to conduct their own aeronautical 
development activities. The quality of development work by NASA 
laboratories is below that of manufacturing companies and service R&D 
companies; however, the quality of aeronautical development work by 
NASA laboratories is better than that of the old NACA and about equal 
to that of other government laboratories. The characteristic quality 
of development work by government laboratories is only average relative 
to the other four aeronautical R&D institutions. However, there are 
additionale rationales for other government laboratories to conduct 
development work in aeronautics. Hence, the development-oriented work 
in NASA laboratories should not constitute nearly so large a portion of 
their total aeronautical activities as is apparently the case. 

Technology advancement .^ NASA laboratories currently seem' to be 
devoting about a third of their efforts in aeronautics to technology 
advancement. The role of NASA laboratories in technology advancement 
should be far smaller than it is at the present time. The only valid 
rationale for NASA laboratories to conduct technology advancement is if 
they have an existing unique capability to accomplish it. The quality 
of technology advancement from NASA laboratories is about the same as 
it was for the NACA laboratories. However, NASA laboratories should 
probably devote even less work to technology advancement than did NACA, 
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unless NASA assumes similar roles. The NASA laboratories characteris- 
tically produce about the same quality of technology advancement as do 
government laboratories in general, which is about average among the 
five aeronautical R&D institutions. The small role of government 
laboratories in technology advancement is justified by additional 
rationale beyond that which applies to NASA laboratories. It therefore 
seems logical that NASA laboratories should have arv even smaller role 1 
in conducting aeronautical technology advancement than do other govern- 
ment laboratories. 

Applied research . Tne primary role of NASA's aeronautical 
laboratories should be to conduct applied research. The nation's 
aeronautics industry and other governmental agencies need more applied 
research from NASA laboratories than the combination of their output in 
all other categories of NASA's in-house aeronautical R&D. The output 
of applied research from NASA laboratories should be roughly doubled. 
Such an increase is necessary in order to bring the supply of NASA's 
applied research into better balance with the relative demand. 
Apparently, NASA's aeronautical laboratories are devoting a far smaller 
portion of their total in-house aeronautics efforts to applied research 
than did the old NACA laboratories. Even the quality of aeronautical 
applied research by NASA's laboratories is lower than that of NACA 
laboratories. 

There is substantial rationale for NASA laboratories to assume 
a prime national role in providing massive amounts of stimulative, 
systematic, and innovative applied research to the aeronautics industry. 
The NASA laboratories should conduct applied research in order to per- 
mit independent evaluation of competitive aeronautical R&D.; In-house 
applied research by NASA provides it the knowledge necessary to con- 
tract aeronautical R&D properly to the private sector. The applied 
research in NASA laboratories provides NASA the knowledge necessary to 
give leadership for coordinating and guiding R&D. The large-scale 
facilities available in NASA can benefit applied research. The NASA 
should conduct applied research where there is a lack of commercial 
incentive. The quality of applied research from NASA laboratories is 
characteristically about equal to that of all government laboratories 
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in general, which tend to produce higher quality of applied research ? 

than any other aeronautical R&D institution. \ 

Basic research . The second greatest role for NASA laboratories '• 

in aeronautics should be in basic research. The basic. research output . > 

of NASA laboratories should be almost doubled in order for NASA to ' j 

play its proper national role in this area. The percentage of basic !; 

| I 

research produced by NASA's aeronautical laboratories- today-seems : to be . , is 

roughly half as large as it was for the old NACA laboratories. The 
quality of NASA's basic research in aeronautics also seems to be lower 
than it was under NACA laboratories. 'i 

There are a number of reasons why NASA should conduct basic j 

research. Basic research by NASA provides a reservoir of aeronautical ; 

knowledge in the public domain. The NASA laboratories should conduct 
basic research in areas where they have the talent. The quality of 
basic research in NASA aeronautical laboratories is about equal to that 
of overall government laboratories, which ranks second only to that of 
universities. The NASA laboratories should do basic research in areas 
where they have unique facilities. Most of the major government-built > 

experimental facilities should remain within government laboratories, 
including within NASA laboratories. The NASA experimental facilities 
would be available to NASA resident research groups. Basic research in 
NASA laboratories facilitates transfer of basic research knowledge into 
applied research. 

' ' i ' 

SPECIAL RECOMMENDATIONS 

The following personal recommendations also are made based on 
an analysis of all the information gained during the course of this . : 

investigation. 

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 

The NASA should consider advocating that it become the central 
aeronautical research agency for the Federal Government. This role for i 

NASA should not exclude highly mission-oriented research efforts by 
other government laboratories. However, NASA could provide an 
efficient central agency for the bulk of the Government's 
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stimulative aeronautical research efforts for both military and civil 
aeronautics. 

Serious consideration should be given in NASA to advocating 
that it also assume a role as a national advisory and coordinating body 
for aeronautical R&D. The rationale for NASA's activities in aeronau- 
tics, as a government laboratory system, would become far stronger if 
such working links to outside technology programs, development pro- 
grams, and aviation operations were established. 

The NASA should consider setting up a special internal manage- 
ment structure for aeronautics similar to that of the old NACA. This 
should be a partnership arrangement with "interlocking boards of 
directors" such as NACA had for over 40 years. It probably will not be 
possible for NASA to serve effectively as the central government 
coordinating, advisory, and research agency in aeronautics unless it 
establishes such a management link with the nation's prime aeronautical 
organizations. 

An investigation should be conducted to examine the role of 
NASA's contract program in advancing aeronautical R&D. This role might 
possibly be different than the role of NASA's in-house work in aeronau- 
tics which was examined during this investigation. 

GOVERNMENT LABORATORIES 

Government laboratories should go to extremes to select work 
that will yield results of maximum value to national aeronautics, as a 
whole. A careful examination should be made of how government labora- 
tories might improve the transfer of full information, on advances 
made under government R&D contracts, to all American aeronautical R&D 
organizations that need it. The government laboratories have substan- 
tial influence in industry and they also have the resources to gather 
information and make it available in a highly usable form. 

ALL AERONAUTICAL R&D INSTITUTIONS 

Every organization in aeronautical R&D should make a funda- 
mental evaluation of its own strengths, weaknesses, natural orienta- 
tions, goals, and relationships to other organizations. Numerous case 
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histories and general studies have shown that poor performance by an 
organization is usually caused by unawareness, neglect, or disregard 
of these basic factors. The data obtained in the current study suggest 
that these fundamental considerations, which seem so very simple on the 
surface, are probably not well understood by many aeronautical R&D 
organizations. 

The tributary technique can be used for many types of funda- 
mental organizational assessments. It is rapid and it reduces the 
expenditure of top management's time. However, this new technique must 
be tailored for each situation. The tributary technique requires sub- 
stantially more preparation and later analysis than do more casual 
conferencing methods. The timing must be precise and every structural 
detail has to be carefully worked out in advance. The key to success 
in using the tributary technique is to pretest every aspect of the 
final procedure. The tributary technique might be constructed to 
examine, successively, such things as an organization's strengths, 
weaknesses, propensities of its people, customer needs, future oppor- 
tunities, goals, and related management factors. 

Advisory group methods for aeronautical R&D can and should be 
improved. For example, all advisory groups should use segment analysis 
techniques. Conference structure methods should also be upgraded to 
fit specific needs. Experience from the current investigation indi- 
cates that it is possible to create conference environments, even in 
highly competitive situations, where advisors can be themselves, 
represent their own interests, and freely provide their best judgments. 


Chapter 2 


INTRODUCTION 

This investigation examines basic rationale for government 
use of different institutions to conduct aeronautical research and 
development. The Government consistently has supported research and 
development (R&D) in the United States for 150 years. Nevertheless, 
the history of government involvement in R&D has been rich in 
controversy (1). The difficulty that the American government has had 
in formulating policy on R&D, in this basically free enterprise 
system, has probably never been more evident than over the last decade. 
The issues historically have fallen into three broad categories: the 

extent of government financing for R&D, the areas of government sup- 
port, and the relative roles of government and private R&D institu- 
tions. This introduction reviews current problems in all three of 
these categories in order to establish a complete framework for consid- 
ering the institutional questions in aeronautical R&D. 

FEDERAL SUPPORT FOR R&D 

There was a total of about $38 billion invested in R&D in the 
United States in 1976 (2:2). For over a quarter of a century, the 
Federal Government has been the source of more than half the funds 

annually invested in R&D in the United States (3:13; 4:2) J. However, 

the federal budgets for R&D have shown a downward trend over the last 

ten years. Federal funding for R&D steadily increased, in constant 

dollars, at average annual compound rates between 5.3 and 14 percent 
from 1953 to 1967 (4:2). The private sector increased its investment 
in R&D at a compound rate of 7.2 to 7.8 percent during the same period 


^Notation (3:13; 4:2) means Reference 3: page 13 and 
Reference 4: page 2. 
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The total national investment in R&D peaked in 1964 at about 3 percent 
of the gross national product (4:3). At that time, the Federal Govern- 
ment provided roughly two-thirds of all funding for R&D. Federal R&D 
funding, in constant dollars, reached its maximum in 1967. The 
federal contribution to R&D dropped from 2 percent to 1. • percent of 
the gross national product between 1964 and 1976 (4:3). The private 
sector continued to support R&D at between 1 and 1.1 percent j>,f the 
gross national product throughout the years from 1955 to 1976 (4:3; 

5:3). Hence, the gradual federal withdrawal from investments in R&D 
over the last decade was totally responsible for a decline of 27 per- 
cent in the annual national investment in R&D to 2.2 percent of the 
gross national product by 1976 (4:2). 

In spite of what seems to be true from the history of civiliza- 
tions, both ancient and modern, economists have not been able to relate 
quantitatively investments in national R&D and technological innovation 
to economic growth. A National Science Foundation report in 1974 
pointed out that 

The impact of technological innovation on productivity *■' 

and economic growth is . . . understood only in general 
terms. Present knowledge of the causal connection between 
innovation and economic returns is not sufficient for 
developing quantitative indicators of the relationship (6:94). 

However, there have been studies on return on investment from R&D for / ' 

companies in specific industries (6:110; 7:25-26; 8) and on the 

influence of R&D on national economic progress (8:2). The results of 

these studies have led most economists to recognize that the impact of /: : I 

R&D on economic growth is quite high (8; 9). : ! 

The value of federal expenditures for R&D are particularly dif- : : 

ficult to quantify in economic terms. Most federal R&D outlays 1 go for / . ? 

defense, space, regulatory activities, and other objectives where the 
primary benefits to society are not reflected in economic quantities. ; 

Two-thirds of the federal expenditures for R&D- in 1976 were for 
defense and space (4:2-3). Other federal R&D programs promote safety, 
health, the quality of life, intellectual growth, and the general 
welfare of the nation. The secondary benefits from technical advances ; 

from federal R&D programs often impact the national economy. However, 
the effects are so pervasive that they are difficult to trace and 
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evaluate. The net effects of most federal R&D on the overall national 
economy is not measurable solely from increased sales and profits for 
the companies conducting the R&D for the Government. This fact has 
apparently led some researchers to assume that effects they cannot now 
directly measure and correlate in economic quantities are. not signifi-- 
cant. These wild assumptions lead to partial economic analyses which 
imply that federal R&Dndoes not warrant the investment (10; 11; 12). ... 
If such fields as science, medicine, politics, philosophy, and the arts 
had waited for economic analysis to prove the worth of every effort 
before attempting to move forward, civilization would probably still be 
in the Dark Ages. 

It is becoming increasingly apparent to wide elements of both 
the public and private sectors that serious alterations have been 
occurring in the relative rate of technological progress and the 
growth of the U.S. economy since the decline of the federal investment 
in R&D (3; 8; 9; 13; 14; 15; 16; 17). America's per capita income has 
moved from first to fifth place in rankings relative to other major 
nations (16:51). The gain in U.S. productivity between 1960 and 1974 
was less than that of four other major industrial nations (6:23-24). 

In fact, America achieved only one-fifth of the increase in productivity 
that Japan recorded. The patent balance of the United States relative 
to other nations fell 30 percent between 1966 and 1973 (6:17). The 
patent balance i. the difference between U.S. patents granted in for- 
eign countries and foreign patents granted in the United States. There 
is little doubt that America's technological and industrial lead has 
been slipping. 

Foreign competi.ti , '~ to U.S. manufactured goods, particularly 
high technology products, has been gaining ground in the world market- 
place (8:1; 17:28). While the United States was cutting back on R&D, 
three other major industrial nations were rapidly and continuously 
increasing the total percentage of their gross national products that 
they were plowing back into R&D for future growth (6:4). West Germany 
and the Soviet Union both had surpassed the United States in the amount 
they were investing in R&D relative to their total gross national prod- 
ucts three years ago. These foreign nations, respectively, spent 2.4 
and 3.1 percent of their gross national products for R&D compared to 
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the 2.3 percent expenditure in the United States. Japan spent about 
1.9 percent of its gross national product for R&D in 1974. However, if 
the previous five-year trend of Japan's increases in investments in R&D 
has continued, then Japan probably surpassed the United States in 
1976 (6:4). Thus, the United States will have to increase its relative 
investment in R&D in order to maintain its competitive position in 
foreign trade. 

There is another possible reason why America's technical inno- 
vations have been slowing down. There are indications that both the 
Government and private industry have been concentrating R&D on 
near-term problems and achieving evolutionary advances, rather than 
attempting to risk more time and effort to create major new innova- 
tions (17). In fact, the average time from completion of an average 
research project to final application dropped from about six years over 
1960-1970 to only three years between 1970 and 1973 (6:79). Great 
innovations tend to take more time to adapt for applications and to 
gain acceptance than do evolutionary advances. Hence, recent trends 
towards "relevance" and requirements for complete financial proofs of 
the value of highly advanced long-term research may well be stifling 
America's continued progress. 

There has been considerable discussion regarding whether the 
Federal Government should increase funding for basic research, applied 
research, or development. Many authors argue that the appropriate 
place for the Federal Government to provide support for the commercial 
sector would be in basic and applied research, rather than for develop- 
ment (for example, 8:38; 18; 19; 20:59). It is argued that there is a 
tendency for commercial firms to invest most heavily in the later 
stages of the R&D process where they can gain competitive advantages. 
Industry is reluctant to invest in long-term research because it is 
risky, difficult to retain for themselves and, therefore, is not as 
likely to yield commensurate profits (3:15; 8:51). Thus, the logic is 
that the Government should invest in research because the returns may 
often be far greater to an entire industry and to society than it could 
be to any single company. The argument goes on to reject government 
support of development activities, except where the Government is the 
real customer for the products (19; 21). The Government is said to 
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lack knowledge of the market and the motivation necessary to yield 
economically viable products. 

There is a school of thought that advocates government support 
of certain development activities. Consider the many well-known 
examples of federal support for development activities in the areas of 
defense, space, housing, transportation, and nuclear energy. An argu- 
ment can be made for government development programs where there 
appear to be very great benefits to society as a whole, but where the 
rewards may not appear great enough to any conmercial firm to warrant 
large, unhedged, private investments. 

The Federal Government continuously faces policy issues on 
exactly what R&D should be done and who should do it. Every year the 
executive and legislative branches of the Government must budget for 
specific R&D programs. The Government must determine the priorities cf 
these programs, based on where the greatest benefits to the nation can 
be achieved. The use of federal laboratories to accomplish a portion 
of this work versus use of the private sector is ar. issue which has 
arisen in many situations dating back to the early 1800's (1; 22). 

AERONAUTICAL R&D 

Aeronautics is a basic industry in the United States. The 
importance of aeronautics to the economy is apparent from some 
relatively simple statistics. Aeronautical products provide the 
highest contribution to export sales of all U.S. manufactured goods and r 
are exceeded only by agricultural produces (23:1). The United States 
has manufactured over 80 percent of all the aircraft in operation by 
airlines throughout the world today (24:20). The net favorable bal- 
ance of trade in aeronautical products in 1974 amounted to about 
$5 billion (6:25-26). Thus, aeronautics is vital to the relative 
strength of the U.S. economy and the soundness of the dollar. 

Aeronautics has helped to alter the entire nature of America's 
commercial and industrial activity. It has enhanced the diversifica- 
tion and geographic dispersion of activities for a huge number of 
American commercial firms, extending them to national and multi-national 
dimensions (25:4-7). It has broadened access to customers and 
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suppliers for large and small firms in every industry. American 
domestic airlines carry about 600,000 passengers a day. This figure is 
expected to increase by 70 percent over the next ten years (26:223). 

One can imagine the tremendous positive effects there must be on the 
economy from the extensive commerce in goods and services conducted 
daily by air travel. Although the effects on the economy have not been 
measured, they may well be more significant to the economy than any 
other factor concerned with aeronautics. However, there is also sub- 
stantial economic activity directly in aviation. Aircraft sales in the 
United States totaled $15 billion in 1975 and airline sales amounted to 
another $15 billion (24; 27:12). Aeronautics currently provides over 
one million direct jobs (23:1) and an undetermined number of secondary 
jobs among a vast array of suppliers and related service industries, 

The United States is highly dependent upon aeronautics for 
defense. In fact, the Department of Defense purchased about 1,400 
aircraft in 1975 at a cost of $7.2 billion, not including purchases of 
armament for these aircraft (24:20,31). Indications are that future 
advances and possibly even increased purchases of military aircraft and 
air-breathing missies will be required in view of relative advances in 
Soviet military might (28:40, 44,64-69). Aeronautics provides a major 
part of the industrial foundation and financial support of the aero- 
space industry, which also produces the nation's missies and space 
products. The aerospace industry's products are as basic to the econ- 
omy, prestige, and defense of the United States as ships were to the 
great seafaring nations of earlier centuries. 

FEDERAL SUPPORT FOR AERONAUTICAL R&D 

Past federal support of aeronautical R&D has been mainly for 
military purposes. The military first employed aeronautics during the 
Civil War. The civilian Balloon Corps was used by the Union army for 
battlefield observation and reconnaissance (1:127-128). The War 
Department supported early experiments on powered aircraft flights 
conducted by Samuel P. Langley around the turn of the century 
(1:284-285). A number of government agencies conducted research in 
aeronautics prior to World War I. However, the National Advisory 
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Committee for Aeronautics (NACA) was founded in 1915 and provided 
focused research, advice, and coordination needed to help boost 
American aviation to the forefront among nations (1:283-287; 29; 30). 

The Government's support of aeronautical R&D for military pur- 
poses gave the United States its world-wide lead in both military and 
civil aeronautics throughout the history of aviation (1; 31:6-27; 32). 
The Federal Government sponsored 45 of the top 51 major technical 
advances made in American aviation prior to 1972 (32:1). The Military 
Services directly sponsored 70 percent of all these advances. Civil 
aviation has relied primarily on military aeronautical R&D in the past ■ 
to provide a technical basis and developments for commercial aircraft. 

Table 1 on the following page shows that federal expenditures 
for aeronautical R&D amounted to about $3 billion in 1976 (33). The 
total government expenditure for aeronautics was only about 60 percent 
of the nation's net export sales of aeronautical products in that year. 
The federal investment in aeronautical R&D amounted to no more than 
15 percent of all the money expended by the Federal Government for 
R&D (4:2; 24:93; 33). The Department of Defense provided 85 percent of 
the Government's investment in aeronautical R&D. 

It is not difficult to understand why the Federal Government 
must supply the bulk of funds for aeronautical R&D. The intensity of 
R&D required to produce the current rate of advances in an industry may 
be measured by the cost of R&D as a percentage of industry sales. The 
aircraft and missies industry expended an average of 21 percent of its 
net sales on R&D between 1961 and 1972 (6:93). This was the highest 
percentage of R&D expenditures of any of the 15 major industry groups 
in the United States. It was more than double the; of the next most 
R&D intensive group, which was the electrical equipment and communica- 
tions industry (6:93). The aircraft and missies industry /itself 
invested 3.5 percent of its net sales in R&D vhich was typical of all 
high-technology industry groups. Thus, one could not expect the air- 
craft and missies industry to provide more of its internal funds for 
aeronautical R&D without substantially raising its prices to its 
primary customers. Commercial sales prices are limited by world-wide 
competitive forces. The Federal Government was the prime customer 
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for missle and space products and for over one-half of the total dollar 
purchases of U.S. r;anufactured aircraft in 1975 (24:31). At' one end or 
the other, the Federal Government, must pay for aerospace R&D to the 
extent of roughly 16 percent or more of industry sales. 

1 Table 1 


Federal Expenditures in 1976 for Aeronautical R&D 
Shown by Performing Institution 


SEGMENT 
OF R 6 D 
COMMUNITY 

WORK CATEGORY 

BASIC 

RESEARCH 

APPLIED 

RESEARCH 

DEVELOPMENT 

ALL SEGMENTS 
(Millions $) 

$66 

$443 

$1837 

MANUFACTURING 
AND SERVICE 
COMPANIES 

15% 

66% 

70% 

NON-PROFIT 

INSTITUTIONS 

2 

2 

2 

UNIVERSITIES 

30 

6 

2 

GOVERNMENT 

53 

26 

24 

TOTAL 

100% 

100% 

100% 


National Science Foundation estimates (33) 


The aircraft and missiles industry is the most R&D-labor- 
intensive industry in the United States. It was the second largest 
employer of engineers and scientists in the United States in 1974, 
which numbered 70,000 that year (5:10). There were about 90 engineers 
for every 1,000 employees in the industry (6:93). This was more than 
twice as large a ratio of engineers and scientists to employees as for . 
any of the other 14 major industry groups in the United Staces. 
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CRISES IN AREONAUTICS 

The aeronautics industry has been encountering serious prob- 
lems. The difficulties are expected to grow more acute in the near 
future unless current trends are reversed by the Government. The U 
Government has let the financial posi tion and the rate of progress of 
this basic industry slip very seriously over the last decade. J\n 
intense national interest in space during the 1960's sharply diverted 
attention away from progress in aeronautics. The effects on aircraft 
R&D programs became even more serious during the 1970's when increased 
national concern over other social and economic problems resulted in a 
general decline of interest in national defense or further advancement 
of America's technological and industrial strengths. The employment, 
of engineers and scientists in the aircraft and missle industry dropped 
28 percent from 1970 to 1975. This was four times greater than the 
average decline of engineers and scientists in other manufacturing 
industries (4:12). Thus, the federal withdrawal from R&D over the last 
decade has hit the aerospace industry the hardest. 

There is a general belief that aeronautics has become a mature 
industry (34:1). The number of new aircraft prototypes greatly 
declined over the last two decades/. Between 50 and 70 new military 
prototype aircraft, large and small , entered flight- testing every five 
years from 1945 to 1960 (34). These figures do not include derivative 
versions of earlier prototypes or any aircraft modification programs. 
Mar\y of the military development programs for transports, helicopters, 
and even light utility aircraft were adapted by commercial manufac- 
turers for civil aviation applications. The space program began in 
1958. The number of new military prototype aircraft entering flight- 
testing dropped, respectively, to 23, 12, and then 13 in each of the 
following 5-year periods from 1960 to 1975 (34). This reduction was 
blamed to a large extent on the escalating costs of military aircraft. 
The reduction in military development programs and possible changes in 
the nature of government-sponsored research may also have reduced the 
impact of federal support on civil aeronautical advancements. “ /Civil 
aircraft manufacturers developed several large wide-bodied jet trans- 
ports specifically for commercial airlines in the early 1970's. These 



engineering developments were based on earlier research and technology 
advancements. However, there have been no new major civil aviation 
development programs started in the last eight years (26:233). 

At the same time, it is clear that many foreign, nations, with 
government support, are continuing to accelerate the pace of their 
aeronautical R&D programs for military and commercial applications 
(23:2-3; 34). Foreign aeronautics industries have clearly stepped up 
their attempts to capture a greater share of the world's aviation 
markets (23:2). The Soviet Union has continued to obtain increasingly 
greater advances in military and civil aircraft (28:40,44,46,47,60, 
62-66; 34). American aeronautics may be approaching "maturity," but 
one might well question how much of this maturity has been caused by a 
failure to provide it sufficient emphasis or to do the right things. 

There will be a huge demand for new civil aircraft transports 
over the next 20 years. Estimates indicate that passenger travel will 
probably grow at a compound rate from 6 to 8 percent per year and will 
therefore roughly double during the next decade (35:6; 36:124). The 
growth of air cargo is estimated at 5 percent annually (36:124). New 
commercial aircraft will be required to meet this increase in demand... 
In addition, the airlines will need replacement aircraft with advanced 
technology to reduce noise, increase safety, cut pollution, and reduce 
fuel costs. Replacement aircraft will be required. The useful life of 
current equipment is estimated to be about 20, years; (36:119). The 
average age of aircraft owned by U.S. trunk airlines plus Pan American 
was calculated to be approximately 7.9 years in 1975 (35:6). It has 
been estimated that U.S. trunk airlines and Pan American will require 
new facilities and equipment amounting to a total of roughly 
$40 billion over the next eight years, $60-65 billion by 1990 (35:6), 
and $162 billion by the year 2,000 (36:119) Thus, the prospects for . 
civil transport aircraft manufacturers would appear bright if it were 
not for financial problems. 

Unfortunately, the U.S. airline industry is not in a financial 
position to raise capital to purchase such massive amounts of. new 
equipment (35:7). The airline industry cannot provide deposits or 
order guarantees to aircraft manufacturers to raise the capital . 
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required for new civil aircraft development programs. The difficulty 
the airlines will have in financing $40 billion of new equipment over 
the next eight years becomes clear when one realizes that the airlines 
earned only about $1 billion in the prior seven years (35:6); This 
industry is already too highly burdened with debt to be a good finan- 
cial risk under current conditions. Twelve major airlines listed on 
the New York and American stock exchanges, which are tracked by Media 
General, had a composite long-term debt-to-equity ratio of 128 percent 
in the 12 months prior to March, 1977 (37:24). This was higher than 
90 percent of 1,500 major common stocks ranked by Media General (37:7). 
Thus, the airlines would not appear to be a good financial risk unless 
there is government support of some nature. 

The Government has already been deeply involved in the down- 
turn in aeronautics. In fact, there are claims that the Government is 
responsible for it (35:8). The profit margins of the airlines are 
influenced by air fares which, are currently controlled through govern- 
ment regulation. Air fares have been allowed to rise only 35 percent 
over the last seven years, while the consumer price ind ex has increased 
86 percent (38:60). Regulated railroad fares were allowed to increase 
100 percent during this same time period (38:60). The average after- 
tax profit margin of 12 major U.S. airlines listed by Media General was 
1.7 percent of sales over a 12-month period prior to March, 1977 
(37:74; 39:6). This ranks below 95 percent of 190 industry groups 
ranked by Media General (37:91). By way of comparison, the composite 
profit margins of eight major regulated electric and gas utility 
groups, ranked by Media General, varied from 5.8 to 24 percent of 
sales. The composite profit margin for 16 major railroads -in the 
Media General ranking was four times higher than for the airlines 
(37:91). The composite return on equity of the 12 major airlines was 
only about 6 percent and the return on total capital was only 2 percent. 
The regulated utility groups had returns on equity and on total capital 
that were roughly two to three times greater than the airline group 
(37:75). Even the railroads had a composite return on total capital 
that was 80 percent greater than the airline group (37:75-76). The 
airlines have reportedly been accused of being inefficient by the head 
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of the Civil Aeronautics Board, but the railroads have not yet been 
termed more efficient (37:91). The airlines currently face the possi- 
bility of g r ''rnment deregulation. They fear this will lead to an 
erosion of air fares under competitive pressures and increased diffi- 
culty in raising capital because of the withdrawal of government 
support (40). 

Aircraft manufacturers are also not in a strong financial 
condition. The large aircraft manufacturers are greatly dependent on 
the Government for most of their income, which is derived primarily 
from military aircraft, missies, and space products. The Government 
holds a monopolistic position relative to these products. It is not 
only the largest customer for these products, but it also controls all 
foreign sales of them. In a market where there are multiple suppliers 
and virtually only one customer, the Government, that customer has a 
very large measure of control over his suppliers and their profits. 

The Government can build or destroy the industry. The composite profit 
margin over a 12-month period prior to March, 1977, for ten major 
aerospace companies listed on the New York Exchange was 2.6 percent of 
sales (37:50). This percentage was less than the profit margins of 
90 percent of the other 190 industry groups that were ranked by Media 
General (37:91). The aerospace firms did have a return on equity that 
was average for the 1,500 companies compared (37:7,50). However, the 
industry faces huge capital requirements. The Milford committee of 
Congress pointed out that "The costs of developing new aircraft and 
engines are approaching the net worth of individual manufacturers, 
seriously inhibiting such projects" (23:2). 

Members of Congress arid executive agencies are currently con- 
sidering ways of increasing government support for U.S. airlines and 
for aeronautical R&D programs (23; 35)., A substantial body of opinion 
favors such action (12; 23; 38; 41; 42; 43:41), but there are also 
opposing viewpoints. Many members of Congress and the executive branch 
favor removal' of supportive price regulation of the airlines (40). 

These viewpoints are based on a Bureau of the Budget study that claimed 
air fares have been too high. The expectation is that lower fares 
would increase airline profits by increasing the passenger volume. 


There is also a contention that aircraft manufacturers have earned 
.' .rates of return on total investment on military aircraft 
projects . . . substantially in excess of the level required to allow 
these firms to attract capital" (18:11). Other opposing arguments are 
that the Government has stimulated the demand side for new aircraft 
through its regulatory powers and tax structure for depreciating com- 
mercial aircraft in use by the airlines (18:12). Arguments have also 
been advanced that the Federal Government should not increase support 
for any R&D and particularly, not for aerospace, because a decrease in 
the demand side should result in a decrease in the federal funding 
supply side (10). There are those who believe that aerospace firms 
should be allowed to fail (10:272-273). Some authors have opposed 
government support to the aircraft and missies industry for R&D based 
on the fact that the extensive R&D required by this industry and for 
the electrical equipment industry, which also receives substantial 
federal support, are not reflected either in commensurate increases in 
financial profits to these industries or increases in their produc- 
tivity (11:217; 12). Most of these opposing arguments are clearly not 
valid in view of the facts discussed earlier. 

REGENT STUDIES ON AERONAUTICAL R&D 

Aeronautical R&D has been receiving more attention and 
increased study during the decade of the 1970's. Recent studies have 
addressed the impact of aeronautics on the nation's economy and wel- 
fare, current problems and future prospects for civil aviation, the 
contribution of military aeronautics to the progress of aviation, the 
Government's role in regulation and support of aeronautics, projected 
future aeronautical advancements, and the potential offered by aero- 
nautical R&D in specific technical areas (23; 25; 28; 32; 41; 42; 44; 
45; 46). These studies recommended that various federal agencies sup- 
port a considerable amount of research, technology advancement, and 
development in aeronautics. A logical question then arises as to who 
should conduct all this work for the Federal Government. This issue 
has already surfaced as an intense topic of discussion in many previous 
publications (for example, 41; 42; 45:Appendix B; 47 and elsewhere). 
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Wide ranges of opinion have been expressed by various aeronautical R&D 
leaders. However, there have been no prior systematic investigations 
of rationale for government use of the various segments of the aero- 
nautical R&D community or to reveal the relative strengths they each 
have to offer. 


AERONAUTICAL R&D INSTITUTIONS - 

How important is it to understand the institutional structure 
and relative strengths of the various segments of the aeronautical R&D 
community? Donald M. MacArthur, Deputy Director of Research and 
Technology in the Department of Defense, pointed out during House com- 
mittee hearings in 1968 that it is not possible to examine even federal 
laboratories in a meaningful way 

* . . unless we place them in proper prospective, with respect 
to the other four types of performers we depend upon in the 
DoD to accomplish our mission. . . . Each of these organiza- 
tional types (industrial firms, universities, non-profit 
insti tutions , and federal contract research centers) has a 
relatively unique, although not mutually exclusive, role to 
play in satisfying DoD requirements (48:3-4). r 

One of his successors, John L. Allen, pointed out in 1974 that 

. . .the differences between in-house laboratories, conmer- 
cial industry, and the academic community lead to differences 
in their capabilities that must be recognized and exploited 

for a maximally effective RDT&E [research and development! 
process (49:3). J 

An economist, Robert Gilpin, stated in a report for the Joint Economic 
Committee of Congress in 1975 that 

... . [The] contrasts and differences among the various 
types of R and D should be primary considerations in the 
development of a national policy towards R and D. They 
should determine the appropriate role of the various 
sectors of the R and D enterprise (university, government, 
and industry). Unfortunately, ... too frequently the 
comparative advantage of each sector has been neglected 
in the fashioning of national policy for R and D (8:36-37). 

Gilpen asserted that part of the reason for Britain's decline in tech- 
nology has been that . . . the British have failed to integrate suf- 
ficiently the three estates of science and technology: universities, 
government, and industry" (8:63). 
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H. Guyford Stever, former Director of the National Science 
Foundation and Science Advisor to the President, pointed out in a 
paper submitted to Congress in 1976 that two of the four things that 
Government must know in order to define an appropriate role of Govern- 
ment in aeronautical research and development are 

... (1) to understand the institutional structure of 

the aviation industry, (2) to understand some charac- . 

teristics of aeronautical R&D .... (50:430). 

Nevertheless, there has been no prior attempt to make a comprehensive 
and systematic assessment of these factors relative to each other. 

Serious questions already have been raised and many actions 
taken in regard to the roles performed by various R&D institutions for 
the Federal Government. They have all affected aeronautical R&D. 
Congressional hearings and investigations in 1969-1970 questioned if it 
was appropriate for manufacturing companies, primarily aerospace 
firms, to receive government support for Independent Research and 
Development (IR&D) (7:41-42). Legislation was proposed to eliminate 
such support. In the early 1970's, many individuals on university 
campuses attacked the role of universities in performing defense- 
related R&D on university campuses. Subsequent actions by many 
universities, and by Government itself, resulted in an alteration in 
the character and extent of such activity in many fields. Proposals 
were made by a Presidential commission in 1970 possibly to have pri- 
vate contractors operate some Department of Defense in-house 
laboratories (51:88). A later study in 1975 recommended that govern- 
ment laboratories should be reduced about 15 percent from their 1974 
levels (47). This recommendation was based on historical funding 
relationships and policy decisions on the roles of government labora- 
tories relative to the private sector. Substantial reductions in 
government scientists and engineers began in 1975. Congressional 
criticisms and pressures from other private firms led to questioning 
the rationale behind the Department of Defense's use of captive 
Federal Contract Research Centers, which are non-profit R&D institu- . 
tions (52). The Department of Defense substantially reduced its use 
of this type of institution in 1976 (52:17). 


The Office of Management and Budget of the executive branch of 
the Government has reportedly been exerting pressure on the Department 
of Defense and the National Aeronautics and Space Administration to 
convert its in-house activities into contracts with the private sector 
and to decrease the number of government personnel within these • 
agencies (53). The policies of the Office of Management and Budget 
have long stipulated that cost should be the major factor in deter- 
mining whether work is accomplished by the Government itself or under 
contract (53; 54). But, is cost the only factor that differentiates 
aeronautical R&D institutions in regard to conducting work for the 
Government? 

DISTINGUISHING FEATURES OF R&D INSTITUTIONS 

Aeronautical R&D institutions may be divided into five 
identifiable categories: manufacturing companies, service R&D compa- 
nies, non-profit institutions, universities, and government labora- 
tories (Appendix B, pages 174 to 176). However, does anything really 
differentiate these types of organizations? After all, they can all 
hire engineers, scientists, technicians, and machinists. An extensive 
body of literature published in recent years shows that organizations 
differ as a result of their internal and external environments (for 
example, 55; 56; 57; 58). The internal environment includes organiza- 
tional structure, management style, work patterns, interactions 
between organizational units, communications, personnel practices, and 
•organizational development. The external environment consists of the 
demands placed on the organizations by products and services required 
by their markets, necessary interactions with customers and suppliers, 
the nature of competition, the economy, governmental actions, community 
relations, and such factors. 

The contingency view of organization and management (for 
example, 55; 59) stresses the importance of obtaining a three-way 
match among the members' propensities, the internal environment, and 
the external environment. It has been shown that different types of 
organizational climates and management methods are required in order to 
conduct different kinds of work effectively. For example. Jay W. Lorsch 
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and John J. Morse v55) found that different management styles and vary- 
ing degrees of feedback were necessary, depending upon the nature of 
the work and the degree of uncertainty inherent in the external envi- 
ronment. In research organizations the external environment is highly 
uncertain. The most effective research laboratories were found to be 
those utilizing less structured organizations, less emphasis on com- 
munications, slower feedback, and more participative management styles. 
The exact reverse was found to be. true for effective manufacturing 
organizations where the external environment was highly certain and the 
nature of the people was different than in the research organizations. 
It therefore requires different types of organizations and management 
methods to best accomplish different types of work. 

The R&D institutions in the United States have developed dif- 
ferent internal organizational climates to fit their situations. 

Studies by Howard M. Vollmer (57), Albert Shapero, and others (58) at 
Stanford Research Institute have revealed extensive categorical dif- 
ferences among the internal environments of government laboratories, 
universities, non-profit institutions, contract research centers, and 
industrial establishments. However, Vollmer (57) presented evidence 
that the internal environments sometimes even differ among the same 
types of R&O institutions in different industries. Thus, it is, 
important to relate roles of R&D institutions to a specific industry, 
because the results may not hold true for other industries. 

There are clearly different external environments for all of 
the institutions engaged in aeronautical R&D. Universities basically 
are engaged in education and must be responsive to the students' needs 
and time schedules. Manufacturing companies depend primarily upon the 
sale of their products and must be responsive to the market and their 
stockholders. Service R&D companies are not dependent upon sales of 
manufactured products and therefore must address their efforts mainly 
to the market for their technical services. They are also under ' 
pressures from their stockholders to produce profits. Non-profit R&D 
institutions have very similar external environments to those of the 
service R&D companies except they do not have to show a profit for 
stockholders. Some non-profit R&D institutions, called Federal 


Contract Research Centers, are partially or totally captive organiza- 
tions of the Government, They are under less competitive pressures 
than other organizations in the private sector. Government labora- 
tories do not operate in situations that are as competitive as those 
for R&D organizations in the private sector. In fact, government 
laboratories control a substantial portion of the funds expended on ..' 
research and technology in the private sector. However, each of these 
groups must coordinate with and be responsive to their customers. The 
possible degrees of coordination and types of interactions with their 
customers are different for the various R8D institutions. Hence, there 
are obvious dissimilarities in the external environments as well as in 
the internal environments of the institutions engaged in aeronautical 
R&D and one might expect that these factors will influence their work. 

PRIOR STUDIES ON' STRENGTHS OF R&D INSTITUTIONS 

There have been no comprehensive systematic studies to examine 
the relative strengths of aeronautical R&D institutions and to examine 
underlying rationale for government use of particular institutions.. 
Vollmer (57) conducted a study in 1965 during which he sampled scien- 
. tists and research managers and obtained a cross-evaluation of the 
relative prestige of several segments of the R&D community. This study 
.also presented data on the willingness of different types of organiza- 
tions to communicate their results outside their own organizations. 

■ Vollmer's investigation sampled the aerospace industry, but it was 
limited to people in the fields of biology, chemistry, mathematics, and 
physics, which were not representative of the major technical disci- 
plines in the field of aeronautics (60). 

In 1970 E. M. Glass reported peer ratings for laboratories in 
the Department of Defense (56:13). The relative performance of these 
laboratories was assessed by comparing the input to the laboratories 
relative to their output. The input to the laboratories was expressed 
in terms of funds, equipment, and property. The output was assessed 
from the peer ratings, papers, and patents. This investigation of DOD 
laboratories did not obtain any comparisons related to any other segment 
of the R&D correnunity. It also did not compare DOD laboratories relative 
to their work in aeronautics. 
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The Department of Defense carried out an assessment of military 
service laboratories from 1974 to 1975, during which it conducted a 
survey only among members of the DOD staff (47 : c-1 to c-12). The sur- 
vey provided data on the staff's opinion regarding whether more or less 
work in specific technical areas should be done under contract or in 
DOD laboratories. It determined the staff's attitude regarding the 
quality of work conducted by DOD laboratories for categories from 
research through development. As part of the investigation', the Air 
Force surveyed 17 internal Air Force "customers" for Air Force labora- 
tory products. Data were reportedly obtained regarding the competence, 
responsiveness, and objectivity of laboratory personnel in relation to 
categorically similar organizations outside the Air Force (47:24). 

The Department of Defense laboratory utilization study presented com- 
prehensive data on personnel, contracting trends, and a very broad 
specturm of Department of Defense activities. This study presented DOD 
rationale for use of various R&D institutions. However, it did not 
assess the relative strengths of all the segments of the R&D corranunity 
as they might be viewed from vantage points outside the Government. 

A Department of Defense task force conducted a study of 
non-profit Federal Contract Research Centers in 1976 (52). Survey 
questionnaires were submitted to nine such centers operating under the 
Department of Defense. The task force conducted interviews with 
people in these centers and in sponsoring organizations. The results 
were discussed in the final report of the task force but the data were 
not reported. Furthermore, this study essentially focused on a single 
segment of the R&D community. The study did not evaluate the roles of 
the other R&D institutions or assess attitudes in the other institu- 
tions regarding their relative strengths for conducting work for the 
Federal Government. 

A NASA committee conducted an interesting study from 1975 to 
1976 entitled "The NASA Outlook for Aeronautics"' (45) . Among many other 
things, it sought to define NASA's mission in aeronautical research, 
technology, and development (45:Appendix B). The conmiittee obtained 
outside viewpoints by personal interviews, letters, and surveys cover- 
ing 38 industrial organizations, 9. leaders in universities, 10 leaders 
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at headquarters level in the Department of Defense, and leaders from H 

6 other governmental agencies in Washington. The NASA correnittee ?j 

interpreted and presented general consensus statements of the comments !; 

■ ■ . ] i 

and also ranges of opinion. This investigation provided useful infor- 

mation on how various other organizations and outside leaders viewed 

NASA's role in aeronautics. The committee then formulated its own j-j 

view of NASA's role in aeronautical R&D. ‘jo 

The earlier investigation into roles for NASA revealed the r.eed ij 

to address the NASA laboratories within the total context of all the 
nation's aeronautical R&D institutions. In addition, it appeared )[ 

necessary to move the inquiry into the realm of quantitative determina- [' 

tions of such subjective judgments in order to gain more specific 
information on the extent and intensity of the opinions of aeronautical 
R&D leaders. It was clear that more highly structured techniques 
would be needed to examine intrinsic strengths, uncover supporting 
rationale, and evaluate the roles of all aeronautical R&D institutions 
relative to carefully defined categories of work. 

CURRENT INVESTIGATION 

The_j.pec-i-f-i.c^purpos e of the current investigation was^to^deter- 
mine what rela t i ons hios-.-J JLan v . exist am ong the relative strengths of 
the nation's aeronautical R&D institutions in regard to conducting 
aeronautical R&D for the Federal Government. It was conceived to pro- 
vide comprehensive, quantitative evaluations of rationale for every 
segment of the aeronautical R&D community, as viewed by leaders from 1 
all of these institutions. Special methodology was devised to surface 
and cross-evaluate relevant rationale. The investigation could not 
logically be confined within the usually desirable constrictions of 
testing a relatively small number of discrete hypotheses. The results 
are therefore extensive in nature. They can be used as a virtual hand- 
book of judgments for specific problems in a wide variety of pertinent 
issues. The information obtained in this investigation should be used 
in formulating government policy on aeronautical R&D. The results also 
can be employed by aeronautical. _R&D organizations, in the public and 
private sectors, in formulating long-range goals and strategic plans. 
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Chapter 3 
METHODOLOGY 

The purpose of this investigation was to determine what rela- 
tionships, if any, exist among the relative strengths of government 
laboratories, manufacturing companies, service R&D companies, 
non-profit R&D institutions, and universities in regard to conducting 
applied research, basic research, technology advancement, and develop- 
ment in aeronautics for the Federal Government. A total of 25 of the 
nation's leaders in aeronautical R&D was selected from among the 
members of the American Institute of Aeronautics and Astronautics. 

Five leaders were selected from each of the segments of the aeronau- 
tical R&D community to provide equal weighting to judgments on 
rationale that supported each segment. Analysis methods were devised 
to reveal the degree of agreement among leaders within each group, 
between specific groups, and for the total sample of all 25 partici- 
pants. 

The investigation was divided into two parts. The first part 
consisted of a survey questionnaire to obtain judgments on factors 
that were hypothesized as being important, lent themselves well to 
written survey questionnaires, and were not appropriate to conference 
discussions. The second part of the investigation consisted of a 
workshop. In the workshop, the participants had the opportunity to 
introduce and cross-evaluate rationale for government use of each type 
of aeronautical R&D institution. The workshop employed a new con- 
ferencing technique that used successive evaluations by homogeneous 
sub-groups, mixed sub-groups, and individuals. The technique was 
devised during this investigation in order to stimulate discussion, 
improve communications, and extract quantitative information on group 
consensus judgments on highly controversial topics. The technique was 
formulated to secure subjective judgments from groups that have 
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intrinsically competitive objectives, unequal power over each other, 
and a degree of mutual dependence. 

DEFINITIONS 

Definitions present a major problem in obtaining subjective 
judgments of the strengths of the various sectors of the aeronautical 
R&D community and rationale that tend to support government use ; of 
those institutions. The various strengths hypothesized in the survey 
questionnaire and those that emerged from rationale obtained during the 
workshop are operationally defined,' respectively, by the questions pre- 
sented in the questionnaire (pages 178-188), the statements made in the 
workshop (Tables 10 through 32), and the related measurements. 

ORGANIZATIONS 

There are three general types of organizations defined in this 
publication which are involved directly or indirectly in aeronautical 
R&D. 

Aeronautical R&D institutions . The terms segments of the 
aeronautical R&D community and aeronautical R&D institution's have 
identical meanings. They refer to organizations that actually conduct 
original aeronautical R&D for the Federal Government. These organiza- 
tions were divided into five relatively homogeneous groups: manufac- 
turing companies (Man Cos), service R&D companies (Serv Cos), 
non-profit R&D institutions (Non-Profs), universities (Univs), and 
government laboratories (Gov Labs). They are defined in Appendix B on 
page 174. The essential differences in the definitions of these insti- 
tutions stem from variations in their principal products, services, and 
ownership structures. It is important to note that the terms govern- 
ment laboratories and NASA laboratories refer only to original, 
in-house, aeronautical R&D conducted by these laboratories. 

Government test centers . The primary tasks of government test 
centers are to conduct ground and airborne tests for other organiza- 
tions. They perform relatively little original aeronautical R&D other 
than developing testing facilities. 
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Government funding organizations . The government funding 
organizations conduct little or no original R&D. They are management 
organizations that provide funds to the performing institutions. 
Appendix B contains a definition of this type of organization on 
page 175. Government funding organizations would include such organi- 
zations as system project offices (SPO's), the Air Force Office of 
Scientific Research, or any. organization that simply! contracts for and 
guides work. However, they would also include organizations, at 
higher government levels, that provide R&D funds to government labora- 
tories. 

ALLOCATION OF EXPENDITURES 

Government contract expenditures that are principally to help 
government laboratory personnel conduct original R&D were attributed 
to the government laboratories (Appendix B, page 174). All expendi- 
tures under government contracts to other institutions were attributed 
to the institutions that conducted the original R&D, even in cases 
where the government laboratories issued the contracts. Government 
funds for Independent Research and Development (IR&D) and the resultant 
efforts by private firms were not included within the scope of this 
investigation. These funds fall under the discretionary control of the 
firms that receive them. These firms use the government funds for IR&D 
in much the same manner as they invest their own internal funds, which 
also were not considered in this Investigation. 

CATEGORIES OF WORK 

The aeronautical R&D process was divided into four categories 
of work: basic research , applied research , technology advancement, 
and development. These categories are defined by the objectives used 
to justify the work and the technical nature of the work. The aeronau- 
tical R&D process actually consists of a continuous spectrum of over- 
lapping activities that extend from basic research through final 
product development. The nature of the work and the. information 
required to do it vary markedly from one end of the spectrum to the 
other. In general, the breadth of the objectives diminishes and the 
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investments increase as activities progress from basic research to 
final product development (4:6; 44:50) J Nevertheless, it is highly 
desirable to create logical definitions of categories of work to 
describe the steps in the R&D process.. They are very useful for man- 
agement, planning, and budgetary purposes. The definitions used in 
this investigation are presented in Appendix B, pages 174 to 176. 

Immense problems and misunderstandings have been created by a 
proliferation of meanings of terms commonly used to divide the R&D 
process into various categories. The American Institute of Aeronau- 
tics and Astronautics published a report in 1975 which contained an 
excellent discussion and comparison of 13 current sets of definitions. 
This publication includes six official sets of definitions used by dif- 
ferent government agencies (44:49-54). Figure 1 illustrates the rela- 
tionships between the current definitions (Appendix B) and two other 
logical sets of definitions frequently used by the Federal Government. 

DEPARTMENT 
OF 

DEFENSE 


NATIONAL 

SCIENCE 

FOUNDATION 


CURRENT 

DEFINITIONS 


RESEARCH RESEARCH AD V, , 



Figure 1: Definitions in the R&D Spectrum 


^Notation (4:6; 44:50) means Reference 4: page 6 and 
Reference 44: page 50. 
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Substantial communications problems result from the many terms -- 
used in Government to describe its categories of R&D. No official 
published set of definitions for the categories of work used by NASA 
for its aeronautics programs could be located for this study. It also 
was not possible to determine from NASA's in-house and contract work, 
conducted under these NASA categories, how the categories related to 
the spectrum of work shown in Figure 1 . A Senate staff report in 1976 
also noted that "It is difficult to gauge the extent of NASA's work in 
basic aeronautics using the current budget breakdown" (53:23). This 
problem is also reflected in much of the literature on NASA's aeronau- 
tics programs which interchangeably use the terms research, 

"research and technology base," "technology," and "R&D." Department of 
Defense officials have also often used the term "technology" inter- 
changeably with “applied research," "R&D," and "development." 
Organizations in both agencies have used hybrid terms to describe 
similar things. Experimental aircraft built just to test aeronautical 
innovations or gain experience in new flight regimes have often been 
called technology demonstrators, hardware demonstration programs, 
concept demonstration programs, demonstration aircraft, experimental 
prototypes, research vehicles, or research aircraft. Here the words 
"research," "technology," and "experimental" often are used inter- 
changeably. From these terms, it might appear that the efforts fall at 
different points in the R&D spectrum (Figure 1). The current set of 
definitions on pages 175 to 176 place such programs well within the 
technology advancement portion of the spectrum and often toward the 
development end of it. Hence, definitions of the categories of work 
present a difficult problem in discussing aeronautical R&D for the 
Federal Government. 

PARTICIPANTS 

A total of 25 aeronautical R&D leaders was selected from. among 
the membership of the American Institute of Aeronautics and Astronau- 
tics. Five participants were selected from each of the five segments 
of the aeronautical R&D community. This selection was made to obtain 
equal weighting of points of view from each of the five types of 
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aeronautical R&D institutions. The participants and their organiza- 
tions are listed on pages vii and viii. Profiles of the participants' 
backgrounds and a description of the author's background are presented 
in Appendix A on pages 166 to 171. 

The following criteria were used by the author in selecting 
each participant: j 

1. Leaders hip- -prime R&D management position in an 
aeronautical R&D institution and/or an outstanding 
reputation for original aeronautical R&D. 

2. Views on subject— unknown to author and sponsor. 

3. Institutional affiliation— equal representation. 

4. Organizational affiliation— representative sample of 
technical products and services. 

5. Geographical location— national dispersion. 

The author was acquainted with only three of the participants prior to 
the study. Their views on the subject were not known. 

There were four problems regarding the representation that may 
have had some influence on the results of this investigation. First, 
the leaders from the non-profit R&D institutions actually represented 
two separate factions. Some leaders came from non-profit institutions 
that were captive organizations to the Government and others came from 
more independent organizations. Observations from the workshop 
revealed some tendency for these two groups to differ because of their 
contrasting situations. Second, one representative in the non-profit 
group came from an organization primarily engaged in space activities, 
although he previously had been known for his work in aeronautics. 
Third, the representation from government laboratories included mem- 
bers of all the major government agencies in aeronautical R&D except 
the Federal Aviation Agency. This omission probably made a difference 
in regard to government civil-aviation R&D activities which deal with 
airline traffic, airports, and safety. Fourth, there were no represen- 
tatives from the helicopter industry, although there was an Army 
representative familiar with helicopter technical problems. 

Some caution is also warranted in regard to possible misinter- 
pretations of the results of this investigation. Most of the partici- 
pants came from particularly high levels of management in aeronautical 
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RSD organizations (pages vii and viii). Their judgments may not neces- 
sarily have been representative of lower level engineers, scientists, 
or managers in aeronautical R&D. For example, Vollmer (57) reported 
considerable differences, in many cases, between viewpoints expressed 
by researchers and those given by research managers. There also could 
have been differences between the participants' judgments and those of 
younger, less-experienced engineers and scientists. The views 
expressed by all participants, as a group, also may not have been 
representative of those that would have been obtained by proportional 
representation from each segment of the aeronautical R&D community. 
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CONVENTIONAL ADVISORY AND CONFERENCE MECHANISMS 

Government and private organizations engaged in aeronautical 
R&D have sophisticated techniques for conducting R&D. However, they 
tend to use conference and advisory group methods that are old, 
simplistic, and often ineffective. The most common method is the 
group "meeting," which approximates the open forum of the ancient 
Greek civilization. In a forum, a group may arrive at a consensus by 
voting in Greek citizen style or the chairman might make a decision in 
the manner of the early kings. There are common hybrid versions of 
these two group decision processes that are commonly used in the aero- 
nautical R&D community. One or two people may prepare written sum- 
maries of group consensus judgments, which might be altered by members, 
of the group or higher authorities. This method does not convey the 
extent of agreement or disagreement among the members of the group. 
Important information can also be lost in the process of interpretation 

Open forums have been plagued by many well-known defects that 
can lead to poor comnuni cations and faulty decisions (61:6,86; 62:1; 
63:177 to 190; 64) . They include domination by -strong eloquent 
personalities,, public conflicts triggering rebuttal -oriented listening, 
embarrassments of publically revising positions, majority pressures, 
interference from irrelevant material, over-politeness, authoritarian 
manipulation by organizers or chairmen, personal perils of free 
speech, pecking order communications, and imprecise methods of 
establishing a group consensus. 







The approaches advocated for improving conference communica- 
tions and group decisions fall into two categories. They either 
attempt to improve human behavior (for example, 65; 66) or they attempt 
to improve the basic structure of the conference method (61; 63;- 64). 
Perhaps the greatest potential is offered by seeking ways to improve 
the conferencing structure since human behavior is extremely difficult 
to change on a large scale. Some prior methods have yielded improve- 
ments in the structure of open forums in appropriate situations. They 
include brainstorming, sub-group discussions, posting, and others (64; 
67). However, many of the major problems associated with open discus- 
sions best can be solved by providing a degree of anonymity. Secret 
balloting has been used to improve decisions at the end of open forums. 
Closed executive sessions have also been used by government advisory 
groups. However, the closed vote at the end of an open forum does not 
aid the early part of the communication process and a closed executive 
session is merely an open forum with more limited participation. 

Written survey questionnaires have been used to provide strict 
anonymity, reduce irrelevant communications, and increase economy (68; 
69). This relatively old and useful method of gathering opinions is 
limited by the fact that it provides only one channel of conmunication. 
It does not allow the participants themselves to exchange information, 
ideas, and viewpoints in order to explore and upgrade judgments. The 
survey questionnaire is also subject to many possible errors in con- 
struction including omission, vagueness, ambiguity, and biased wording. 

The impersonal surveyrquestionnaire technique provided the 
foundation of the Delphi method which was invented, not by the early 
Greeks, but by the Rand Corporation around 1950. The Delphi method has 
hundreds of variations but they all basically provide for anonymous 
communications by the use of successive rounds of questionnaires (61). 
After each round, the participants receive the results summarized in 
statistical form and perhaps written information from the responses. 

The participants may then modify their views after rethinking the situa- 
tion. This procedure has been found to lead towards a convergence of 
group opinions. The Delphi method has been criticized severely for its 
high participant dropout rates, tendency to produce conformance to ' 
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majority opinion, lack of proof of validity, and loss of group creativ- 
ity that might be possible through personal interactions (67). There 
were other criticisms attributed to the Delphi method that were no more 
than incorrect applications of analysis methods, sampling procedures, 
statistical techniques, and reporting requirements (67). They were not 
fundamental to the basic characteristics of the Delphi method as a con- 
ferencing technique. 

SEGMENT ANALYSIS 

The methods used in this investigation were constructed to pro- 
vide quantitative information on the points of view of the leaders from 
each of the segments of the aeronautical R&D community and the combined 
opinions of all 25 participants. In a criticism rr past use of expert 
opinion, H. Sackman suggested that all future metr-,:.s should employ 
techniques that allow identification of sub-groups and analysis of 
their opinions (67:26). Sackman cited prior studies for the Government 
where extensive group biases were suspected, but which were difficult 
to trace because of the methods employed. Many people in Government 
expect advisory groups to be above biases. One published study 
reported that advisory groups composed of members of certain Air Force 
aeronautical R&D organizations were completely objective in giving 
advice in a Delphi study (70). This study supposedly showed that they 
even provided judgments contrary to the goals of theif own organiza- 
tions. The proof given left some room for doubt because the impacts 
on future organizational goals were not carefully identified and the 
statistical significance of the comparisons was not computed. 

The current investigation employed segment analyses for several 
reasons. It was hypothesized that leaders from the same types of aero- 
nautical R&D institutions would tend to have common views regarding the 
relative strengths of all segments of the aeronautical R&D community. 
Another hypothesis was that many of these views would be shared by 
leaders from all the aeronautical R&D institutions. If the segment 
analyses showed this to be true for certain opinions, then these view- 
points might be accepted as postulates. However, aeronautical leaders 
from the different institutions were also expected to disagree on many 
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of these issues. In these cases, the segment analyses would provide 
the opportunity to select the set of "advisors" that seemed most 
likely. to have made the best judgments. In addition, the sub-group 
analyses would reveal sub-group viewpoints that might explain reasons 
for different attitudes that one might encounter'in dealing with people 
from these institutions in the future. 

SURVEY QUESTIONNAIRE • 

A survey questionnaire was prepared and mailed to all 25 aero- 
nautical leaders prior to the workshop. The questionnaire is given in 
Appendix C, pages 178 to 188. It included institutional strengths that 
were hypothesized as being significant but which did not lend them- 
selves well to discussion at the workshop. The survey questionnaire 
started the participants 1 thinking about the problem and the related 
definitions. However, the results were not fed back to the partici- 
pants nor was the survey discussed during the workshop. The intent was 
to limit the impact of the questionnaire on the workshop. The partici- 
pants completed matrices on funding in the questionnaire and at the end 
of the workshop (Appendix C, pages 187 and 188). 

The questionnaire was prepared in about two months. It was 
completely rewritten seven times and was pre-tested on six engineers 
a secretary, and a housewife. The estimated time to complete it was 
45 minutes. A total of 22 participants returned the completed ques- 
tionnaires by mail before the workshop, as requested. These question- 
naires were anonymously returned, identified only by segment of the 
aeronautical R&D community. The other three participants personally 
handed in the completed questionnaires at the start of the workshop. 

WORKSHOP AND TRIBUTARY TECHNIQUE 

The tributary technique was devised to provide a conference 
structure for the workshop. The technique was designed to establish a 
flow of information somewhat similar to the way that tributaries 
develop from their own sources, successively interact with other tribu- 
taries, mix in rivers, and flow into the oceans. The technique was 
devised for conferences where the participants belonged to organizations 
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that did have an overall common objective, where there were various 
degrees of mutual dependence among sub-groups. However, the technique 
was most specifically designed for situations where these groups had, 
at the same time, strongly diverse, conflicting, and often competitive 
objectives. It was for high-threat environments where many partici- 
pants might initially feel that individual free speech could adversely 
impact their own organizations. The technique was devised for diffi- 
cult topics that required exploration, examination, and highly subjec- 
tive judgments by qualified people. It was for topics that required 
clear consensus judgments by groups, rather than possibly divergent 
individual innovations. 

PRINCIPLES OF TRIBUTARY TECHNIQUE 

The tributary technique was based on nine main principles 
related to group dynamics. 

Interesting group discussions . The tributary technique was 
devised for group discussions combined with impersonal communications, 
as opposed to the completely impersonal approach used in. the Delphi 
method. The reportedly massive dropout rates .in the Delphi process are 
indicative of frustration or a loss of interest (67:52). In such 
cases, one cannot control the type or extent of participation. The 
current method was devised to create an exciting atmosphere. There is 
reason to believe that people think and work best under such conditions. 
Prior studies indicated that small groups, of about six members, with 
internally free discussions were most likely to reach consensuses 
(64:193). This concept was found to be valid, independent of the skill 
of the leaders. These prior studies indicated that periods of 20 to 30 
minutes for intense group discussions were best, although' some discus- 
sion methods dealt with brief topics in only 6 minutes (64). 

Highly structured steps . The tributary technique provided a 
very highly structured conference. In fact, the conference structure 
was used to replace the role usually assumed by a chairman. Neither 
the author nor the sponsor had any control whatsoever over the par- 
ticipants' activities within the framework of the conference structuie. 
The expectation was that the conference structure would also increase 
participant teamwork, reduce interference from extraneous material, 



\ ’ - 

\ - .. . 

54 

increase efficiency, and increase the participants’ satisfaction. Some 
earlier studies indicated that groups are, in general, more satisfied 
with conferences that have a good deal of procedural structuring, or 
control, than they are with meetings that are less orderly (64). 

Graduated tasks . The entire process was divided into a series 
of graduated tasks in order to help the participants progressively make 
more difficult decisions. The tasks and the goals were carefully 
defined for each set of discussions. The results at the end of each 
stage were self-contained and logically complete. The interaction 
process was planned so that every group's output, at every stage, was 
rapidly reviewed by a peer group. The members of each group had the 
ability and the power. to make decisions and complete their tasks at 
each stage of the process. Earlier studies in group dynamics indicated 
that group productivity increases when the participants know that they 
have the power to resolve the problem that they are given (64). 

Rethinking and feedback . The procedures were devised to give 
each participant up to three chances to change decisions. A partici- 
pant could alter his decisions each time without anyone else realizing 
that he was revising his position. The rethinking process was also 
designed to contribute to the participants' thought processes and to 
allow them to' move on to increasingly difficult decisions. This proce- 
dure was expected to yield a convergence in viewpoints. There was some 
prior evidence from a Delphi study that just rethinking decisions 
played a strong part in the convergence process that took place in the 
Delphi method (62). Feedback was also used in the .tributary technique. 
All conference methods make headway by providing feedback in some form. 
The tributary technique was structured to yield timely and complete 
feedback on sub-group consensus viewpoints. 

Time pressure . The time for each step purposefully was set at 
a level that would create pressures on the groups. Studies at, the 
University of Michigan on group dynamics have shown that "The more 
urgent the problems, the more productive the group" (64:63). 

Graduated confrontation . The conference was structured to 
limit direct confrontations between participants from dissimilar 
groups, until all homogeneous groups had an opportunity to carefully 
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consider all viewpoints. A series of steps provided indirect communi- 
cation between homogeneous groups prior to mixed group meetings. 

Constructive competition . The idea was to channel the natural 
sense of competition into constructive contributions. The tributary 
technique was based solely on positive inputs followed by quantitative 
evaluations of those inputs. Each sub-group was to be challenged, by / 
a sense of competition, to improve on the inputs of other groups. / 

Anonymit y. The intent was to increase the freedom to conrsun'i-/ 
cate by submerging individual judgments in homogeneous sub-group / 
opinions. The maximum threatening exposure was in mixed group meetings 
that followed homogeneous sub-group meetings. However, in the mixed 
group meetings, each participant had the option of impersonally stating 
viewpoints as the opinions that came out of his homogeneous group 
meetings. . 

Quantitative output . The tributary technique was devised to 
provide complete quantitative output on group consensus judgments at 
the end of every phase of the process. Groups and individuals had to 
make quantitative decisions which provided focus and task-oriented 
behavior. 

TECHNIQUE DEVELOPMENT AND PREPARATIONS 

The tributary technique was designed in conceptual form. The 
process was planned for five homogeneous sub-groups considering four 
categories of work. The entire workshop was to be limited to a single 
day. 

Test groups . A model of the tributary technique was devised 
to test the whole process by using two homogeneous groups and two cate- 
gories of work. The model workshop was conducted with five partici- 
pants from one government laboratory and five participants from a 
service R&D company. The participants in the test groups were inter- 
viewed, as a group and individually, after the model workshop. 

Information was obtained from the test groups on the time required to 
complete the tasks, the clarity of the instructions, communications 
difficulties, individual attitudes, and the efficiency of the proce- 
dures. The tributary technique and its implementation were substan- . 
tially modified based upon this information. 
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Training the workshop observers . The day before the conference 
five observers were instructed on their functions. Each group discus- 
sion session in the conference was to have one observer and one 
recorder. The observer's functions were to enter formulations of ten- 
tative group viewpoints on a blackboard. co»mient on clarity of the 


statements, transfer group decisions to a flip chart, keep the group on 
time, and interpret procedures. The observers were instructed riot to 
participate in the discussions in any manner nor attempt to act as 
meeting chairmen. They were shown how to handle potential difficulties 
that had arisen in the earlier test-group sessions. They rehearsed all 


procedures during practice sessions. The observers were as follows: 


Stephen H. Achtenhagen 

Professor 

School of Business 

San Jose State University 

William Brickr.er 

Professor 

School of Business 

San Jose State University 

John R. Chirichiello 
Associate Director 
Industry Studies 
National Scier.ce Foundation 


Stewart E. Fliege 
Professor 

School of Business and Management 
Pepperdine University 

Dennis D. Schiffel 
Group Leader 

Effects of Public Policy on S&T 
National Science Foundation 


None of the observers had ever worked in the field of aeronautics nor 


had they ever had any affiliations with orgar ‘zations engaged in aero- 
nautical R&D. 


OPENING SESSION 

The agenda followed during the workshop is given in Appendix C 
on pages 189 to 191. The author took 10 minutes of the opening session 
to review and discuss a list of definitions whicn were distributed as 
handouts (Appendix B, pages 174 to 176). The roles of the observers in 
the sub-group sessions were explained. The participants were informed 
that no judgments would be traceable to individual.'. It was considered 
very important not to reveal the entire process in advance which might 
have interfered with sequential decisions and biased group actions. 

Only the procedures and tasks for the homogeneous sub-group sessions 
were outlined at the opening session. 
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The participants were divided into five " ib-groups. Each 
sub-group consisted of representatives of a single segment of the 
aeronautical R&D community. The homogeneous sub-groups met in 
separate rooms. 

Rationale development: self-justification . Each sub-group was 

asked to indicate, on a flip chart, as many as four primary reasons why 
the Federal Government should use its type of R&D institution to con- 
duct each category of aeronautical R&D. In other words, each sub-group 
would submit a maximum of 16 self-generated statements of rationale why 
the Government should use its type of institution to conduct all four 
categories of work from basic research through development. Each 
statement of rationale was to be a sentence that contained a single 
reason. The statements of rationale did not have to be unique to just 
one institution or a single category of work. The same rationale 
could be applied to more than one aeronautical R&D institution and to 
multiple categories of work. Tentative statements were first formu- 
lated on a blackboard. The observer transferred a final statement to 
the flip chart when three or more of the five sub-group members agreed 
with it (for example, see Appendix C, page 192. The recorder then 
entered the rationale statements on a hand-held form (Appendix C, ■' 
page 193. The time allotted for the first session was 1 hour. 

Rotation, review, and expansion of rationale . The sub-groups 
then sequentially rotated rooms for each of the following sessions. 

The same room was always used for consideration of rationale related to 
the R&D institution whose representatives had originally occupied the 
room. The observers, recorders, and flip charts remained in their 
original rooms. Hence, each reviewing group read the rationale state- 
ments left on the flip chart by the prior groups. The first task 
assigned to each review group was to enter on the flip chart, new 
statements that either supported, clarified, or corrected rationale for 
government use of the R&D institution that had initially occupied the 
room. The review group was not permitted to physically alter the pre- 
vious statements appearing on the chart or to enter negative state- 
ments. The process used by each reviewing group in formulating, 
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agreeing upon, and recording the new rationale was exactly the same as 
had been used in the initial session. Next, the reviewing group again 
examined the previous rationale to determine if it considered ‘any of 
the rationale to be completely invalid. When four out of five members 
of a review group agreed upon such a judgment, the recorder then noted 
it by placing a blackball (a) on the hand-held rationale sheet 
(Appendix C, page 193). The recorder also noted which homogeneous 
group entered each statement of rationale by using a plus (+). This 
information did not appear on the flip chart and was not available to 
the reviewing groups during these homogeneous group sessions. There 
was only one round of rotation. Therefore, none of the homogeneous 
sub-groups had an opportunity, during these sessions, to see any of the 
new rationale entered on its original flip chart. The single-round 
approach was used in order to avoid concerns of competitive counter- 
actions that might have followed the first round of reviews. However, 
the one-round approach meant that none of the sub-groups could evaluate 
all of the new rationale entered by all reviewing groups on the flip 
charts of every other RAD institution. The time allowed for each 
review session was 25 minutes. The observers indicated that perhaps 10 
additional minutes should have been allowed for the last session 
because of a substantial accumulation of rationale statements. 

FEEDBACK ON RATIONALE 

The rationale statements and judgments from all the prior homo- 
geneous sub-group sessions were next distributed to all participants. 
This distribution was accomplished by reproducing the accounts main- 
tained by the recorders on the hand-held rationale sheets (Appendix C, 
page 193). The participants were next given instructions, in a general 
session, for the mixed group sessions which followed. 

MIXED GROUP MEETINGS 

The participants were then divided into five mixed groups. 

Every mixed group had one person from each of the five segments of the 
aeronautical R&D community. Four of these groups each considered a 
different category of work: basic research, applied research, tech- 

nology advancement, or development. Each mixed' group reviewed all of 
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the rationale statements and group judgments that pertained to work by 
all five aeronautical R&D institutions within the one work category. 

The fifth mixed group dealt with the entire spectrim of work 
conducted by NASA laboratories. This group considered all rationale 
and group judgments that applied to work conducted by government 
laboratories. 

Rankings of rationale . The mixed groups first ranked, in order 
of importance, the rationale statements supporting a particular cate- 
gory of work by each segment of the aeronautical R&D community. The 
rankings applied only to a single segment and no attempt was made to 
cross-evaluate rationale for the different types of institutions. The 
group that considered NASA evaluated the rationale for government 
laboratories relative to NASA laboratories. The group members ranked 
the rationale in order of importance within each category of work. 

There was no attempt made to cross-rank the importance of rationale for 
different categories of work by NASA. 

The rankings for all groups were from 1 (highest) to 5 or 6 
(lowest). Statements could also be rated below the numerically ranked 
items by leaving blanks. A mixed group could indicate that statements 
were invalid when four out of the five members agreed to it (©) ; The 
other rankings had to be approved by three or more members of a mixed 
group. A mixed group could also add new statements of rationale as had 
been done in the earlier sessions. 

Distribution of work . The mixed groups then formulated quanti-. 
tative judgments of how deeply each segment of the aeronautical R&D 
community should be engaged in each given category of aeronautical R&D 
for the Government. Each group arrived at a consensus regarding a 
reasonable percentage distribution of federal expenditures among the 
various segments of the aeronautical R&D community for each category of 
work. These determinations were not meant to represent specific recom- 
mended numerical constraints. They were to reflect roughly the 
expected extent of the relative roles of the various institutions in 
view of the rationale that had just been considered. The mixed group 
that considered NASA laboratories indicated their judgments on the 
percentage distribution of NASA laboratory funds that should be 
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distributed among in-house basic research, applied research, technol- 
ogy advancement, and development. The time allotted for the mixed 
group sessions was 1 hour and 20 minutes. 

INDIVIDUAL EVALUATIONS AND FEEDBACK 

A general session was then held for all participants. The 
author read each mixed group's rankings of rationale. The participants 
duplicated these rankings onto their copies of the hand-held rationale 
sheets. This means of feedback probably could have been handled more 
efficiently by reproducing and distributing these tables. The partici- 
pants then considered all prior judgments and individually ranked the 
relative importance of the rationale statements in the left-hand column 
of thtir own hand-held rationale sheets (Appendix C, page 193). The 
author then displayed the mixed group's work distribution. The par- 
ticipants tl.on individually entered their own evaluations of relative 
work distribution on special tables provided to each individual 
(Appendix C, page 199). This entry was made for each of the four cate- 
gories of work and for NASA laboratories. The time allotted for this 
phase was 1 hour and 25 minutes. 

An extremely high dropout rate occurred in filling out individ- 
ual evaluation forms in the very last portion of this phase, when NASA 
laboratories were considered. Immediately, 8 participants dropped out 
and this number steadily increased to 17 dropouts. Only one dropout 
departed from the conference table. This sudden cessation of activity 
may have been caused by the laborious nature of information feedback 
and handwritten individual evaluations. On the other hand, it may also 
have been caused by concerns that the sponsor (NASA) might be able to 
trace individual judgments. Such concerns were later expressed to the 
author by several participants. At the end of the workshop, the par- 
ticipants were instructed to code their written evaluation sheets in a 
manner that clearly guaranteed anonymity. It would have been better if 
the coding had been done at the beginning of this session, than to post- 
pone it to the end. 
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SUMMARY OF TRIBUTARY TECHNIQUE 

Figure 2 shows a summary of the tributary process. The initial 
questionnaire was not actually part of the conferencing technique since 
it was not used in the workshop. 


! Questionnaire. Individual ! 
, Allocation of Funds ; 



Figure 2: Summary of Tributary Process 


TRIBUTARY TECHNIQUE EVALUATICN 

At the end of the worksnop, the participants completed ques- 
tionnaires aimed at determining their initial reactions to the 
tributary technique. The results are shown in Appendix C, pages 
201-202. Eighty-five percent of the participants stated that this 
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workshop methodology improved communications over their experiences 

with methodologies used in previous advisory groups where the topics 
% 

were highly subjective and the advisors had naturally diverse group 
interests. These participants estimated that they, as a group, had 
taken part in over 350 categorically similar meetings. The one par- 
ticipant who felt that there was no improvement over prior methods had 
taken part in about a dozen categorically similar conferences. >. 


QUANTITATIVE METHODS 

There were essentially three different types of quantitative 
data taken during this investigation; rank order numbers , percentage 
distributions, and opinion ratings . The first two types of data were 
averaged for each group and are reported in that form in Chapter 4. 

The third type is reported without alteration. The average values 
reported for the rank order data are called group consensus rankings 
or just consensus rankings. The average values reported for the 
percentage distributions are simply called group percentage distribu- 
tions. 

It is extremely important to know the probability of chance in 
regard to the participants' judgments. Variations in agreement among 
judgments by different members of each group (that is, the scatter in 
the data) are reflected in statistical probabilities of chance. These 
probabilities are reported along with all average val^-e data. They 
indicate the statistical probability that the difference between com- 
pared mean values could have occurred by chance. A small value for the 
probability of chance (P) means that the reported differences between 
any two or more average values given by a group were probably due to 
strong agreement among the members of the group rather than just a 
chance occurrence. 

GROUP CONSENSUS RANKINGS 


The group consensus rankings were calculated by scoring the 
participants' rankings, averaging the scores, and then re-ranking the 
items in the same order as the averages. 

Rankings and related computations . Appendix C-5 describes the 
methods used for ranking, re-scoring the rankings, and then calculating 
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the group consensus rankings. Group consensus rankings will be the 
best estimates of the true overall group rankings if they are statist]- > 
cally significant (71:100-102). In other words, they must reflect a 
low probability of chance. 

In considering rankings, it is important to realize that the 
judgments do not necessarily correspond to the indicated scale interval. 
For example, there can be a much greater difference in judgments 
between items ranked first and second than between the second ranked 
item and an item ranked third. 

Agreement among participants . Measures of the extent of agree- 
ment among members of the groups, for the group consensus rankings, 
were calculated using two related indicators: the Kendall coefficient 

of concordance (W) and the associated probabilities of chance 
(71:94-106; 72 : 229-239) . The Kendall coefficient of concordance is 
simply an index. It equals a value of one when there is perfect agree- 
ment among the judges and it approaches close to zero when there is no 
agreement. The Kendall coefficients were corrected for ties awarded to 
different items in a ranking by the same judge. The correction for 
ties tends to increase the values of the Kendall coefficients.. 

The probabilities erf chance (P) associated with the Kendall 
coefficients were also computed for all group consensus rankings 
(71:94-106; 72:235-237). The probability of chance was not corrected 
for ties in accordance with accepted methods. It was assumed that the 
Kendall coefficients and the group consensus rankings were statisti- 
cally significant when the probabilities of chance were less than or 
equal to 5 percent (a < 0.05). Interesting things occurred when the 
participants highly agreed on close to the same rank for almost all 
items, creating mostly ties. The Kendall coefficients went up because 
of the correction for ties. This index indicated that the members 
tended to agree. However, the probabilities of chance also increased 
| which correctly indicated that the differences between group consensus 

I rankings for the items became statistically insignificant. 

I I The statistical probability of chance was computed for certain 

\ rankings using the x 2 *test (72:175) or the binomial test (72:36-42). 

| ; These tests were used when there were insufficient variables to employ 

; Kendall coefficients and the related significance tests. In addition, 

i ■ ' 

i 

i 

t 

i- ** “® . ' ”* 1 ■' "■V'- 1 ■ " " - — — —— ' -I 1 r-. LI- I. I 1 — — . MU 

. t , • un i — 





the x 2 ~test was used in some important comparisons to evaluate the sta- 
tistical significance of changes in ranks for a particular iten, from 
one set of circumstances to another. This methodology is covered in 
greater detail in the discussion of the results. The results presented 
in Chapter 4 show the probability of chance (P) and the statistical 
test used in the computations, i.e. W, x 2 » or reference to the binomial 
distribution. • : 

GROUP PERCENTAGE DISTRIBUTIONS 

The questionnaire and workshop called for individual partici- 
pants to render judgments that involved distributing percentage alloca- 
tions among a number of items. The percentages were always required to 
total 100 percent when sunned up over all the items in a distribution. 

Mean values . The group percentage distributions were computed, 
by averaging the percentages allocated to any one item by all members 
of a group. This averaging was done for each item in the distribution. 

Agreement among participants . There were two statis ical tests 
used to determine the extent of agreement between members of a group in 
regard to the mean values. They were the F-test and the t-test. These 
statistical tests apply to opinion data which reflect interval scaling 
in judgments. Both of the tests yield the probabilities that the indi- 
cated differences in mean values might have occurred by pure chance. 

The F-test determines the probability of chance for the indicated dif- 
ferences in group mean values for an entire percentage distribution 
(73:223-230). The t-test is a method used for small samples to deter- 
mine if the differences between only two mean values are statistically 
significant (73:191-198). The probability of chance (P) was computed 
using the t-test for two related samples (74:386). The statistical 
probabilities were determined directly from the F distribution and the 
t distribution using a Texas Instruments SR- 52 calculator, a Hewlett- ■ 
Packard HP-65 calculator, and their associated statistical program 
packages. The results given in Chapter 4 indicate F and t values and 
identify which statistical tests were used for comparisons between 
various mean values. 



Chapter 4 

RESULTS AND DISCUSSION 

The discussion of results is divided into three' sections. The 
first section deals with factors that were hypothesized as distin- 
guishing the various segments of the aeronautical R&D community from 
each other in regard to conducting work for the Federal Government. 

These results were obtained by use of the survey questionnaire. The 
second section of this chapter presents the rationale developed during 
the workshop by the aeronautical leaders from the various R&D institu- 
tions. The third section presents the results on the distribution of 
work among the various aeronautical R&D institutions for activities 
ranging from basic research through development. 

HYPOTHESIZED DISTINGUISHING FEATURES OF 
AERONAUTICAL R&D INSTITUTIONS 

This section covers the results obtained from the survey ques- 
tionnaire submitted to the participants prior the the workshop. j 

i 

QUALITY OF WORK j 

' I 

Table 2 presents the aeronautical leaders' group consensus j 

rankings on the quality of work characteristic of various R&D institu- 
tions. The institutions being ranked are shown at the top of the table 
and the affiliations of the judges are indicated in the left-hand col- 
umn. Hence, it is possible to read across any row and determine how 
the aeronautical leaders from each institution ranked the quality of 
work by all aeronautical R&D institutions. The group consensus rank- 
ings are from 1 (highest) to 5 (lowest). Each group consensus ranking j 

was calculated by averaging the ranks given to each institution by all j 

the judges in a group and then awarding group consensus ranks to all | 

the institutions in the same order as the averages (pages 62 to 64). I 
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The Kendall concordance coefficient in tfie right-hand column indicates 
the degree of agreement among the rankings awarded by the judges affil- 
iated with the group shown in the left-hand column of the table. Per- 
fect agreement would yield a Kendall coefficient of 1 and no agreement 
would produce a coefficient close to 0 (see page 63). High Kendall 
coefficients, such as shown in' Table 2a, indicate that the individual 
judges in each group ranked the relative quality of basic research very 
close to the values shown in the table. 

The total group- consensus rankings shown in the bottom rows of 
Table 2 were computed from the individual rankings made by all 25 
participants. The total group-consensus ranking is an equally weighted 
composite of the judgments of the aeronautical leaders from all of the 
institutions. There was statistically significant agreement, among, the 
total group of leaders, on the rankings of work quality ever the full 
spectrum of R&D. Agreement was assumed to be statistically significant 
if there was not more than a probability of 5 percent that the indi- 
cated comparisons between values could have occurred by chance. In 

Table 2 

Group Consensus Rankings on Quality of Work 

Responses to ; ". . . rank the various segments of the aeronautical R and D x 
community In order of the relative overall QUALITY of aeronautical R, T and D - 
work performed. . . . Rank the organizational segments from 1 (highest quality) 
to 5 (lowest quality) 


Table 2a: Basic Research 


^''''''rankings 

BY+ 

CF+ 

MAN 

COS 

SERV 

COS 

NON- 

PROFS 

UNIVS 

GOV ! 

UBS 

KENDALL 
CONCORDANCE 
COEFFICIENT (W) 

MAN COS 

(S) 

4 

5 

3 

1 

2 

0.81 

SERV COS 

(5) 

5 

3 

? 

1 

4 

0.78 

NON-PROFS 

(5) 

5 

4 

2 

1 

2 

0.74 

UNIVS 

(S) 

5 

4 

2 

1 

2 

0.85 

GOV LABS 

.m. 

5 

4 

2 

1 

2 

0.71 

ALL 25 
PARTICIPANTS 

5 

4 

2 

I 

2 

0.70 
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Table 2c: Technology Advancement 



Table 2d: Development 



Over 95% probability jn cases not asterisked that ranking by each group reflected 'a oeaningful 
group discrioination of differences in ranks, rather than chance variations (P < a ■ 0.05). 
Asterisk (*) indicates this probability is under 9S\. -p 









other words, there had to be at least a probability of 95 percent that 
the consensus values represented a meaningful overall group discrimina- 
tion between ranks. The Kendall coefficients indicate that the extent 
of agreement among all the participants was the highest for basic 
research and development, which represent the two outer extremes of the 
R&D spectrum. The extent of agreement among all participants, as a 
group, was lowest for applied research. 

The total group-consensus rankings in Table 2 indicate that the 
aeronautical R&D institutions produce qualities of work that complement 
each other across the R&D spectrum. The institutions that characteris- 
tically produce the highest quality of work in each category were con- 
sidered to be universities for basic research, government laboratories 
for applied research, and manufacturing companies for both technology 
advancement and development. The total group-consensus rankings for 
the universities reflect their orientation towards fundamentals, rather 
than more applied pursuits. The non-profit R&D institutions were 
recognized as supporting the prime institutions with quality work 
because they received the second highest places for both basic and 
applied research. The more applied supporting roles of the service R&D 
companies were apparent from the second place consensus ranks they 
received for the quality of technology advancement and development. 

The quality of wcrk by manufacturing companies was considered high for 
technology advancement and development, but low for basic and applied 
research. This finding suggests that manufacturing companies are most 
oriented towards creating competitive,.. new, near-term products. 1 They do 
not seem so well oriented towards research for far-term innovations in 
aeronautics. The government laboratories were the only institution that 
received total group-consensus ranks that were within the topthree 
places for the quality of work over the entire spectrum of all R&D 
activities. This finding implies that government laboratories have a 
broader orientation than do other aeronautical R&D institutions. 

There was significant internal agreement on 75 percent of the „ 
sub-group consensus rankings. The aeronautical leaders from the 
universities and government laboratories were the only two sub-groups 
that displayed significant internal agreement on all consensus rankings 
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of all four categories of work by all the institutions. There was also 
remarkably close agreement between these sub-group consensus rankings 
and the overall consensus rankings by all participants. . 

Biases in sub-group self-rankings can be evaluated by examining 
the diagonal of numbers extending from the upper left to the lower 
right of every table. A bias becomes apparent when a rank on this 
self-ranking diagonal is better than ar\y of the ranks awarded by peer 
sub-groups. This bias can be seen by comparing a rank on the diagonal 
with values up and down a column from the diagonal. A total of 
45 percent of the sub-group consensus self-rankings was better than any 
comparable rankings by peer sub-groups. None of the self-rankings were 
below peer group rankings. It is also interesting to note that ... 
three-fourths of the sub-group self-rankings were either 1 or 2 for all 
categories of work. Every sub-group displayed some natural biases in 
ranking the quality of their own work, except the university leaders. 
Hence, there were natural biases when the sub-groups ranked the quality 
of work by their own institution. However, a careful comparison of 
sub-group rankings indicates that the total group-consensus ranking, in 
the bottom rows of Table 2, balanced out individual sub-group biases. 

QUALITY OF AERONAUTICAL R&D FROM NASA LABORATORIES 

The National Aeronautics and Space Administration (NASA) 
replaced the National Advisory Corrmittee for Aeronautics (NACA) in 
1958. Numerous recommendations have been made that the Government 
should support all R&D, including aeronautical R&D, in a manner charac- 
teristic earlier of NACA (8:53; 23:48; 42:58; 45:Appendix B:9, 11, 19, 
29). 1 Table 3a presents the first of several comparisons that were 
aimed at exploring what basic differences exist, if any, between NACA 
and NASA laboratories in regard to in-house aeronautical R&D. Table 3a 
shows the number of aeronautical leaders that ranked the quality of 
aeronautical R&D by NASA's laboratories either below, equal, or above 
that of the old NACA. 


Notation (8:53; 23:48; 42:58 and so forth means Reference 8: 
page 53 and Reference 23: page 48 and Reference 42: page 58 and so 
forth. 
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All comparisons in Table 3a were statistically significant. 

The bottom row in this table shows the number of aeronautical leaders 
that would have been expected to rank NASA respectively below, equal, 
and above NACA, if the rankings had been made by completely random 
selections under the probabilities of chance for infinite size samples. 
When 2 numbers are selected at random and each of them may range from 
1 to 5, there are 5 ways that the 2 numbers could turn out equal 
(P=0.20) and 20 ways that they could be unequal (P=0.80). There is the 
same probability (P=0.40) that the second number could be higher or 
lower than the first number. Hence, one would expect that out of every 
25 random selections of 2 numbers there would be 5 equal numbers, 10 
lower second numbers, and 10 higher second numbers relative to the 
first numbers selected. 

The results in Table 3a indicate that the aeronautical leaders, 
as a group, believed that the quality of basic research and applied 
research in aeronautics by NASA laboratories is below that characteris- 
tic of the earlier NACA laboratories. The aeronautical leaders, as a 
group, believed that the quality of NASA's laboratory work in aeronau- 
tical technology advancement is about the. same as it was for NACA. 
However, the participants indicated that the quality of NASA's in-house 
development work in aeronautics is superior to that of NACA. " 

Table 3b presents a comparison of the quality of * ironautical 
R&D by NASA laboratories relative to all government laboratories. The 
results indicate that the aeronautical leaders, as a group, perceived 
no differences between NASA laboratories and other government labora- 
tories in regard to the quality of any category of work. This conclu- 
sion is indicated either by (a) the number of aeronautical leaders that 
gave equal ranking to both NASA and overall government laboratories, 

(b) an equal number of leaders ranking NASA both above and below 
overall government laboratories, and (c) a lack of statistical signifi- 
cance in rankings. 

UTILITY OF RESULTS 

An initial hypothesis was that the aeronautical R&D leaders 
would weigh the usefulness of the results produced by the various insti- 
tutions differently than quality of their work. The results in Table 4 
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Table 3 

Quality of Aeronautical R&D by NASA Laboratories 
compared wi th 

Earlier NACA Laboratories and Overall Government Laboratories 
(Displays number of leaders making indicated ranking) 


R esponses to : "Similarly rank the relative QUALITY of in-house aeronautical worJc characteristic of Just 
HA5A/NACA. Use a scale from 1 (highest quality) to 5 (lowest quality) relative to the segments of the 
aeronautical R and D ccrasnunity shown in the previous matrix. . . . KASA/HACA may rank equal to, 
higher or lower than the overall government laboratory segment." 


Table 3a: NASA (1976) versus NACA (1958) 


QUALITY 

^-v^FROM 

WORK 

NASA 

BELOW 

NACA 

NASA 

EQUALS 

NACA 

NASA 

ABOVE 

NACA 

PROBABILITY OF CHANCE 

X 2 

PROB. fP) 

BASIC RESEARCH 

. 

17 

6 

2 

11.50 

0.3% 

APPLIED RESEARCH 

14 

7 

4 

6.00 

5.0% 

TECHNOLOGY 

ADVANCEMENT 

S 

12 

8 

12.70 

0.2% 

PRODUCT 

DEVELOPMENT 

3 

7 

IS 

8.20 

1.7% 

EXPECTED FROM 
PURE CHANCE 

IQ 

5 

10 


Table 3b: 

NASA versus Overall Government Laboratories 

QUALITY 

^"■^FROM 

WORK 

NASA 
BELOW 
GOV LABS 

NASA 
EQUALS 
GOV UBS 

NASA 
ABOVE 
GOV LABS 

PROBABILITY OF CHANCE 

X 2 

PROB. (P) 

BASIC RESEARCH 

8 . 

9 

8 

4.00 

* 13.6% 

APPLIED RESEARCH 

9 

12 

4 

13.50 

• 

0.1% 

TECHNOLOGY 

ADVANCEMENT 

S 

11 

9 

9.80 

0.7% 

PRODUCT 

DEVELOPMENT 

11 

• 

7 

7 

1.80 

* 40.?% 

EXPECTED FROM 
PURE CHANCE 

10 

5 

10 

■ 
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Table 4 (Continued) 


. WORK 
CATEGORY 

CONDUCTED 

BY 

m^ESm 

UTILITY 

EQUALS 

QUALITY 

UTILITY 

ABOVE 

QUALITY 

PROBABILITY OF CHANCE 

X 2 

PROB, CP) 

TECHNOLOGY 

ADVANCEMENT 

MAN COS 

5 

13 

7 

16.20 

Below 0.1% 

SERV COS 

7 

15 

3 

25.80 

»l 

NON-PROFS 

6 

17 

2 

36.80 

If 

UNIVS 

4 

18 

3 

42.30 

II 

GOV LABS 

3 

15 

7 

25.80 

II 

PRODUCT 

DEVELOPMENT 

MAN COS 

0 

25 

0 

100.00 

II 

SERV COS 

4 

18 

3 

42.30 

II 

NON- PROFS 

3 

19 

3 

49.00 


■EBBHH 

2 

22 

1 

72.30 

II 

GOV LABS | 

3 

17 

5 

36.20 

II 

EXPECTED FROM PURE CHANCE 

10 

5 

10 

■■ 
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indicate that the participants did not differentiate between utility 
and quality. Hence, no further comparisons for the utility of R&D are 
presented in this publication. 

ABILITY TO FOCUS ON SPECIFIC REAL PROBLEMS 

Table 5 presents group consensus rankings on the characteristic 
ability of the aeronautical R&D institutions to focus on achieving 
solutions to specific real problems. 


Table 5 


Ability to Focus on Solving Specific Real Problems 
Group Consensus Rankings 

Responses to : "All other things being equal, rank each of the following types of 
organisation In order of their ability to focus on achieving solutions to 
specific real problems. . . . from 1 (most able) to 5 (least able) for each type 
of organization." 


RANKINGS ^ 
BYF 

^OF+^ 

MAN 

COS 

SERV 

COS 

NON- 

PROFS 

UNIVS 

GOV 

LABS 

KENDALL 
CONCORDANCE 
COEFFICIENT (W) 

MAN COS 

(5) 

1 

3 . 

2 

5 

4 

0.52 

SERV COS 

(5) 

1 

2 

3 

5 

3 

* 0.34 

NON-PROFS 

(5) 

1 

2 

3 

5 

4 

0.65 

UNIVS 

(5) 

1 

1 

3 

5 

3 

0.65 

GOV LABS 

(5) 

1 

2 

3 

5 

3 

0.54 

ALL 25 
PARTICIPANTS 

1 

2 

3 

5 

4 

0.42 


Over 95% probability in cases not asterisked that ranking by each group reflected a meaningful 
group discrimination of differences in ranks, rather than chanc? variations (P < a » 0.05). 
Asterisk (•) indicates this probability is under 95%, 


There was significant agreement on the total group-consensus ranking 
shown on the bottom of the table. The total group-consensus ranking 
indicates that manufacturing companies were believed to have the 
greatest ability to focus on solving specific real problems. The manu- 
facturing companies were followed in order of ability by service R&D 
companies, non-profit R&D institutions, government laboratories, and 
universities. It is interesting to note that the government labora- 
tories were ranked fourth in this comparison even though they were 
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ranked first on the quality of applied research shown in Table 2b on 
page 67. This infers that the government laboratories were considered 
to produce the highest quality applied research, but^they did not seem 
so effective in bringing their talents and resources to tear on solving 
specific real problems as are most institutions in the private sector. 

The sub-group consensus rankings were not markedly different 
than the total group-consensus ranking. Only the sub-group from ser- 
vice R&D companies, internally,- did not significantly agree on their 
consensus rankings. The government laboratories were ranked between 
third and fourth by all sub-groups. Even the leaders from the universi- 
ties ranked their institution last in regard to the ability to focus on 
and solve specific real problems. 


DISSEMINATION OF RESEARCH AND TECHNOLOGY 

There are two types of results that occur when the Government 
expends funds on research and technology. The primary results appear 
at the end of the effort in terms of a required output, such as a docu- 
ment, a computer program, or hardware. The primary output often enters 
the public domain under most government contracting procedures for 
research and technology. However, there is also a secondary result in 
the form of an increased capability and retained body of knowledge that 
remains within the organization that performed the work. This incre- 
ment in internal capability and captured knowledge can give performing 
organizations a competitive advantage over other organizations. The 
Government may indeed wish to build up residual capabilities and stores 
of knowledge in aeronautical R&D’organizations. However, the Govern- 
ment may also wish to have this publically funded knowledge and capac- 
ity openly accessible for purposes that will serve all American avia- 
tion. The staff of the Senate Committee on Aeronautics and Space . 
Sciences expressed the following concern in regard to aeronautical 
research: 

We question whether scientists and engineers in private indus- 
try who are exploring new basic technology under contract to 
NASA are truly accessible to engineers in competing com- 
panies . . . . Thus, we are concerned that heavy reliance on 
out-of-house contracting for basic aeronautical research will 
inhibit healthy competition in the industry's appl ication of 
that technology (53:23). 

ORIGINAL PAGE IS 
OF POOR QUALITY 
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Table 6 presents group consensus rankings on the willingness of 
the various institutions to provide outside organizations with full 
information on government-funded research and technology, even beyond 
the information contained in published reports required by the Govern- 
ment. Development purposely was not considered in this assessment 
because competition in even government-funded development programs is 
considered desirable in the free enterprise economic system. Competi- 
tion naturally leads to restrictions in disclosures between competitors. 
However, the Government usually expends funds on aeronautical research 
and technology advancement specifically for the purpose of advancing 
the overall aviation industry. 

There was remarkably high agreement among the total group of 25 
aeronautical leaders in regard to the relative willingness of different 
types of institutions to fully disclose the results of government- 
funded research and technology. The total group-consensus rankings 
indicate the following things. The universities were considered the 
most willing to provide full information. They were followed in 
descending order of willingness by government laboratories, non-profit 
R&D institutions, service R&D companies, and finally, manufacturing 
companies (Table 6). It is interesting that this was the exact 
reverse of the total group-consensus rankings of the characteristic 
ability of these institutions to focus on and solve specific real 
problems (Table 5). The results in Tables 5 and 6 thus imply .that the 
greatest competitive inclinations tend to exist on the part of institu- 
tions whose internal capabilities are most directly oriented toward 
solving problems associated with specific applications. 

The results shown in Table 6 are consistent with earlier inves- 
tigations. Vollmer found that research managers sampled in industry 
indicated that it would be cause for dismissal if a researcher gave 
research ideas to outsiders without clearance from his own management 
(57:61). However, research managers sampled in universities, govern- 
ment laboratories, and non-profit R&D institutions indicated that 
giving research ideas to outsiders would not be adequate grounds for 
dismissal in those institutions. The evidence obtained on publication 
rates and the influence of publications on salaries of engineers and 
scientists in these institutions all indicated that open publication 


was more frequent and was valued to a greater extent in universities, 
government laboratories, and non-profit institutions, than they were in 
manufacturing companies. 

It is interesting to note in fable 6l that even the leaders from 
the manufacturing companies ranked their own institution last. The 
leaders from the service R&D companies ranked their own institution 


next to last. There was strong internal agreement in these sub-grr-ips 
on these rankings. The sub-group consensus rankings were also close to 
the total group-consensus rankings by all participants. 

Table 6 


Willingness to Disseminate Research and Technology 
Group Consensus Rankings 


Responses to : "Rank the following types of organization in order of their 
willingness to provide outside organizations with, FULL INFORMATION on government 
funded RESEARCH and TECHNOLOGY, even beyond -the '-’information contained In published 
report 8 required by the government. . . .write one number In each box, from 1 
(most willing) to 5. (least willing) for each segment of the aeronautical 
community." 


'^RANKINGS 
BY 4 

_0F->^ 

MAN 

COS 

SERV 

COS 

NON- 

PROFS 

UNIVS 

GOV 

LABS 

KENDALL 
CONCORDANCE 
COEFFICIENT (W) 

MAN COS 

.(5) 

S 

4 

3 

1 

2 

0.80 

SERV COS 

(5) 

S 

m 

3 

1 

1 

0.58 

NON-PROFS 

.(5) 

5 

4 

2 

I 

2 

0.90 

UNIVS 

(5) 

S 

4 

3 

1 

2 . 

0.89 



5 

4 

3 

1 

1 

0.95 

ALL 25 
PARTICIPANTS 

5 

■ 

3 

■ 

2 

0.79 

1 


Over 95\ probability in all cases that ranking by each group reflected a meaningful 
group discrimination of differences in ranks, rather than chanco variations. 


EXPERIMENTAL FACILITIES 


Figure 3 indicates that the aeronautical leaders from all R&D 
institutions quite strongly believed that major aeronautical experi- 
mental facilities built by the Federal Government should be managed by 
organizations that would provide usage and/or results to the entire 
aeronautical community. 
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Opinion on atatement : "Major aeronautical experimental facilities built by the 
government should be managed by organisations that will provide usage and/or 
results to the entire aeronautical community." - 


MAN COS 

( 5 ) 

1 

- 


1 

III 

SERV COS 

( 5 ) 


1 

.. 


Jill 

NON- PROFS 

( 5 ) 


1 

1 

III 


UNIVS 

( 5 ) 



II 

1 

II 

GOV LABS 

( 5 ) 

1 



1 

III 


1 1 

1 

III 

II 

1 

I 


-4 -3 

-2 

-1 0 1 

2 

3 

4 


strongly 

disagree 


neutral 



strongly 

agree 




Figure 3: Usage of Major Experimental Facilities and Results-. 

Figure 4 shows the aeronautical leaders from all R&D institu- 
tions were substantially of the opinion that resident research groups 
should be located at major experimental facilities, if these facilities 
can be useful for basic or applied research. 


O pinion on statement : "There should be resident research groups located at major 
s«rv.tnutlca.l experimental facilities built by the Federal Government, If these 
f rcllltlfe* ca~ oe useful for basic or applied research." 
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(5) 
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Figure 4: Location of Research Groups at 

Major Experimental Facilities 
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Figure 5 indicates the aeronautical leaders' views on whether 
government or private institutions can most effectively 7 manage and con- 
trol major experimental facilities for aeronautical R&D. The sub- 
groups from the manufacturing companies and service R&D companies were 
almost evenly divided on this issue. The leaders from non-profit R&D 
institutions believed that institutions in the private sector could 
most effectively manage and control these facilities/ The leaders from 
government laboratories and universities both believed the Government 
could most effectively manage and control major government-built 
experimental facilities. 


Opinion on statement ; "Organizations in the private sector can more effectively 
manage and control major government -built facilities than can the government." 
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Figure 5: Most Effective Institutional Control of Major 
Government-Built Experimental Facilities 


Table 7 presents the aeronautical leaders' final judgments on 
the organizations that actually should have custody and managerial con- 
trol of major aeronautical experimental facilities built by the Federal 
Government. The institutional affiliation of each group of judges is 
shown in the first column at the left of the table. The top row of the 
table displays the types of organizations that would receive a percent- 
age of the total capital value of the major experimental facilities 
built by the Federal Government. The figures in the columns under each 
type of institution show the average percentages of the total capital 
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value of all such facilities that were awarded to that organization by 
each of the sub-groups shown in the column on the far left of the table.' 
The values in each row should total 100 percent except for small round 
off errors. The distribution of average percentages among the various 
institutions, along each row, is termed the group percentage distribu- 
tion of experimental facilities. The column of numbers on the far 
right of Table 7 gives the probability that the group percentage dis- 
tribution in each row might have occurred by chance. 

Table 7 

Distribution of Major Experimental Facilities 
Group Consensus Percentage Distribution 

(percentage of total capital value of all 
major, government-built, aeronautical, 
experimental faci 1 i ties — 
over $1,000,000 each) 


Responses to : "Roughly Indicate the percentage of the capital value of all major 
aeronautical experimental facilities built by the government that you think should 
be located at and under the managerial control of the following types of 
• organ! rat ions." 


^DISTRIBUTION 

FOR 

MAN 

SERV 

NON- 

UNIVS 

GOV 

GOV 

TEST 

PROBABILITY 
OF CHANCE 

BY4- 


COS 

COS 

PROFS 


LABS 

ORG 

F 

PROS. (P) 

MAN COS 

(5) 

3 % 

6 % 

3 % 

8 % 

47 % 

33 % 

9.50 

P<0.1% 

SERV COS 

(5) 

0 

20 

6 

9 

26 

39 

2.90 

3 % 

NON- PROFS 

(4) 

12 

15 

12 

9 

28 ' 

24 

*1.63 

20 % 

UNI VS 

(5) 

7 

5 

12 

10 

38 

28 

8.38 

P<0.1% 

GOV LABS 

(5) 

3 

2 

10 

7 

57 

21 

7.05 

P<0.1% 

all 

RESPONSES 

(24) 

5 % 

10 % 

8 % 

8 % 

40 % 

29 % 

23.57 

P<0.1% 


There was significant agreement among all participants or. the 
total group percentage distribution shown on the bottom row of Table 7. 

The total group decision favored placing the largest share of the 
experimental facilities under the Government (69%) . This could be a 
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reflection of aeronautical leaders' concern over general accessability 
by their organizations to the facilities and the results from these 
facilities. The aeronautical leaders distributed more of the major 
experimental facilities to government laboratories (40%) than to 
government test centers (29%). This result reflected their interest in 
having resident research groups at these facilities. 

There was significant internal agreement in all the sub-groups 
on the percentage distribution of experimental facilities, except for 
the sub-group from the non-profit R&D institutions. Non-profit R&D 
institutions was the only sub-group that also believed that the privatp 
sector could more effectively manage and control major experimental 
facilities built by the Government (Figure 5). The various sub-groups 
differed by factors of two or more in regard to some of the percentages 
of experimental facilities allocated to certain institutions. The 
differences are easily reconciled by taking self-oriented biases into 
account and neglecting the statistically non-significant results. In 
fact, the group consensus distribution for the total group apparently 
balanced out most of the biases. 

TECHNICAL GUIDANCE ON CONTRACTS 

The policy of the Office of Management and Budget (0MB) has 
been to contract a- r ' , »ch government work as possible to the private 
sector rather than having it. done by the Government itself (54). 

Assume for a moment that all aeronautical R&D was contracted to the 
private sector without regard to any other factor. First of all, the 
Government would still have to select and plan paths to advance aero- 
nauti cal R&D. It would still have to determine +: .e best technical pro- 
grams to pursue. These decisions involve resolving technical issues 
regarding both what is needed and what is feasible. The Government- 
would still have to select the best technical projects from among many 
proposals. It would have to continue to seek contractors, evaluate the 
technical merits of competitive proposals, select the best proposals 
and most competent contractors for the wcrk, provide the contractors 
with technical guidance toward the desired goals, insure that the 
technical progress of the work remained on track, provide additional 
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Technology advancement (T) is the systematic adaptation and use 
of knowledge to achieve the capability of practical Industrial applica- 
tions, the expansion of the engineering and Industrial arts, and 
advancement of the mechanisms for product development and production. 
This includes expansion of basic engineering data, methods, techniques, 
basic componentry, tools, and experience needed to be able to develop 
and produce products. This process sometimes requires major large- 
scale hardware Integration steps, including building and testing 
"bread boards" of components, "technology dsnonstration" hardware, and 
so-called “proof-of-concept" hardware. Technology advancement can be 
.properly and successfully justified on the basis of being necessary 
/preparatory steps to achieve needed, or Improved, product developments 
or manufacturing methods. 

/ Development (D) is the study, design, engineering, testing, 
tooling, and construction that are necessary prior to, and for the 
purpose of, obtaining an end product to be manufactured. It includes 
; : the development of end-item components and subsystems. The development 
process results in one or more prototype test articles from which 
' evolves the item to be manufactured. 
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QUESTIONNAIRE 


-' School of Business and Management >' £.',vlv;t : “^v£a.' 


Pepperdine University 

Aeronautics. Advisory Project 
770 Welch Road 
Suite 154 

Palo Alto, California 94304 


V- «- 


; V.. . ;./ • AERONAUTICS ADVISORY SURVEY ■ v.K-'.-, ■ 

FEDERALLY SUPPORTED AERONAUTICAL R, T and D • . 

-O; -V,; I. INTRODUCTION ' ..•> .- '-f y*iy? '-’i? 

In this questionnaire we request some fairly general -idgments, - 
advice and opinions regarding the conduct of federally supported 
aeronautical R, T and D by the various segments of the aeronautical 
r and D community. For the purpose of this questionnaire. Federal ' 
support of the private sector will mean all government expenditures, 
direct and indirect, associated with contracts and grants to the private 
sector for original R, T and D. These funds may come from any Federal 
source, including government laboratories. On the other hand. Federal 
support of and related expenditures by government laboratories will 
refer to all government funds, direct and indirect, (or ORIGINAL 
IN-HOUSE R, T and D conducted by government engineers and scientists 
from these laboratories. This includes all Federal funds expended for 
salaries, overhead, facility constructions, instrumentation and all 
support service contracts associated with government laboratory 
personnel conducting in-house work. 

II GENERAL INSTRUCTIONS 

The questions ‘hat follow will require considerable thought based 
on your general background and experience. At times, you may feel that 
your background is insufficient to give firm answers. Nevertheless, we 
would appreciate it if you would fully answer all of the questions as 
best you can. We expect to gain considerable insight from the sun total 
of all the responses because the cumulative experience of all the 
participants will be quite broad. The relative degree of uncertainty 
the participants may have in regard to any questions will be properly 
reflected by the scatter of the data, that is the variation in the 
answers received from all participants as a group. Therefore, it is 
important that you completely answer all questions. 

Some questions ask you to make a single selection from among a 
group of answers, where more than one may seem to apply. Please select 
the .answer that seems most applicable, but never give more than the 
number of answers requested. 
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I 7. TECHNICAL DISCIPLINE [Check below only one prime topic area, 

j i.e. check one box) 

■;[ Systems technology Propulsion, continued 

□ Vehicle design, studies Q Rocket propulsion systems .. -., 

j and analyses □ Fuels and propellants " 

□ Effectiveness and safety □ Combustion 

.! 0 Support Systems Mechanics and Control J.’ 

-! ' 0 Flight Testing □ Flight mechanics and Control 

Atmospheric sciences □ Guidance and control 

Aero acoustics Q Avionics 

D Atmospheric environment Structures and Materials 

! 0 Fluid and gas dynamics □ Structures 

; j 0 Thermophysics □ Structural dynamics 

. O Plasmadynamics □ Materials 

Propulsion □ Other 

Air breathing prop. (write in) 

systems ■ 

DEFINITIONS OF CATEGORIES OF R, T and D WORK 

Read over all the following definitions of categories of- 
aeronautical R&D work. These definitions will be used throughout 
j- the questionnaire; therefore, it is very important for you to be 

j completely familiar with them. You will also need them to answer 

I the next questions about yourself. 

BASIC RESEARCH (R) - is the SCIENTIFIC pursuit of fundamental 
KNOWLEDGE in a general technical area, independent of known applica- 
tions. The results consist of new theories, basic experimental 
observations (data), '••‘.J understanding of physical nature. The 
knowledge gained is expected to be of such interest or value within the 
! technical area, that the work can be properly and successfully justi- 

I . fied without being tied to specific known applications. / 

j i APPLIED RESEARCH (R) - is also the SCIENTIFIC pursuit’ of fundamental 

i KNOWLEDGE, but it is aimed toward restricted types of applications or a 

j particular application. The results consist of theories, systematic 

| experimental data (which may be quite extensive and detailed) and under- 

J - standing. Applied research solves the basic problems and opens the way 

I for the technological advancements needed to achieve the desired 

: ' applications. Applied research is properly and successfully justified’ 

primarily by expected advances relative to such applications. ,.V 

TECHNOLOGY ADVANCEMENT (T) - is the systematic adaptation and use 
of knowledge to achieve the capability of practical industrial applica- 
tions, the expansion of the ENGINEERING and INDUSTRIAL ARTS, and 
advancement of the MECHANISMS for product development and production.; 

This includes expansion of basic engineering data, methods, techniques, 
basic componentry, tools and experience needed to be able to develop 
and produce products. This process sometimes requires major large-scale 
hardware integration steps, including building and testing "bread boards" 
of components, "technology demonstration" hardware and so-cal led 
"proof-of-concept" hardware. Technology advancement can be properly . 
and successfully justified on the basis of being necessary preparatory 
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steps to achieve needed, or improved, product developments or 
manufacturing Methods. 

DEVELOPMENT fDl - the s tudy, design, engineering, testing, 
tooling, and construction that are necessary prior to, and for the 
purpose of, obtaining an end product to be manufactured. It includes 
the development of end-item components and subsystems. The development 
process results in one or more prototype test articles from which 
evolves the item tolbe manufactured. 

YOUR CATEGORY OF WORK 

8. Refer to the preceding definitions of R, T and D. Check the 
kind of aeronautical work below that you are primarily associated with 
in your current position (check one). 

□ Basic Research O Development 

□ Applied Research □ Other 

□ Technology Advancement (Write in) 

9. Refer again to the preceding list of definitions of R, T and 
D. Check the type of work below that is the primary business of your 
company, division, or organization (check one). 

□ Basic Research G Development 

D Applied Research G Other • 

□ Technology Advancement (Write in) 


DEFINITIONS 

OF 

SEGMENTS OF THE AERONAUTICAL RfiD COMMUNITY 

Read over all of the following definitions of organizations 
involved in aeronautical R, T and D. These definitions of the various 
segments of the aeronautical R and D community are used throughout the 
questionnaire. It is important to be familiar with all of the defini- 
tions. Questions about them will follow. 


MANUFACTURING COMPANIES (MAN COS) - commercial concerns that 
primarily manufacture products as their major source of gross income 
and profits. 

SERVICE R5D COMPANIES (SERV COS) - commercial concerns' 
that principally provide original R, T or D and other non-manufacturing 
services for other companies, government agencies or the public to 
obtain its primary source of gross income and profits. Service may 
extend from research to product development, such as may be performed 
by research associations, consulting firms and small RgD companies. 

NON-PROFIT RSD INSTITUTIONS (NON-PROFS) - institutes and 
institutions that primarily perform R, T and D services for companies, 
the government or the public--without opportunity for profit, 
dividends or capital gains by owners. This segment includes separate 
institutes affiliated with universities and non-profit subsidiaries 
of profit companies. 
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UNIVERSITIES (UNIVS) - colleges, technical institutes, technical 
schools or universities where education is recognized as the central 
function of the organization, and within which other professional activi- 
ties in R, T and D exist to a greater or lesser extent than education. 

GOVERNMENT LABORATORIES (GOV LABS') - all in-house government 
laboratories where original aeronautical R, T or D is performed. Such 
laboratories may also contract with organizations in the private sector 
. for R, T and D to a greater, or lesser extent than the in-house work. 

For the purpose of this survey, work by government 1 laboratories refers 
strictly to in-house work and direct expenditures by government 
laboratories related to in-house work. 

GOVERNMENT KSD FUNDING ORGANIZATIONS (GOV FUND ORGS) - 
government organizations that distribute technical work among other 
groups, monitor the work and provide funding for contract or in-house 
government R, T or D done by other organizations. Government R6D 
funding organizations do little or no in-house R, T or D of their own 
v; in comparison to their funding activities. But, they may be involved 
in considerable technical monitoring, study, direction, and guidance 
activities. 

' YOUR AFFILIATION 

10. Referring to the previous list of organizational definitions, 
check the type of organization below that best describes the principal 
nature of your company, division, institution or government organization 

□ Government Lab 

□ Government R6D Fund Org 

D Other . 

(Write in) 

IV. JUDGMENTS, OPINIONS AND ADVICE 

11. QUALITY OF WORK 

In the following matrix, rank the various segments of the 
aeronautical R5D community in order of the relative overall 
QUALITY of aeronautical R, T and D work performed. Proceed down 
each column in the matrix and enter one number in each box. Rank 
the organizational segments from 1 (highest quality) to 5 (lowest 
quality) for the type of work indicated at the head of the column. 

See definitions given on pages 3-5. 


G Manufacturing Company 

□ Service R5D Company 

□ Non-profit RSD Company 


QUALITY OF WORK 



GOV LABS 


12. Similarly rank the relative QUALITY of in-house aeronautical 
work characteristic of just NASA/NACA. Use a scale from 1 (highest 
quality) to S' (lowest quality) relative to the segments of the 
aeronautical RSD community shown in the previous matrix. NASA/ 
NACA laboratories fall within the general category of government •- 
laboratories. Nevertheless, NASA/NACA may rank equal to, higher or 
lower than the overall government laboratory segment. 

QUALITY OF WORK BY NASA/NACA 


| Basic 
i Research 


Applied 

Research. 


Technology 

Advancement 


Development 


NACA (to .*• . "• ~~ 

l 958 ^' I . 

13. . USEFULNESS UK RESULTS l 

In the following matrix, rank the various segments of the 
aeronautical R$D community in order of the relative overall 
USEFULNESS of the RESULTS they produced. Proceed down each column 
in the matrix and enter one number in each box. Rar.k the results 
produced by each organization in order of USEFULNESS from 1 (greatest 
usefulness) to 5 (least usefulness) for the type of work shown at the 
head of each column. 
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USEFULNESS OF RESULTS 
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2 


.3 


1 ; 


4 


5 


14. Similarly rank the relative overall USEFULNESS of the in-house 
aeronautical work characteristic of just NASA/NACA. Again, use a 
scale from 1 (greatest usefulness) to S (least usefulness) relative 
to the segments of the aeronautical community shown in the previous 
matrix. 


USEFULNESS OF NASA/NACA RESULTS 



Basic 

Research 

Applied 

Research 

Technology 

Advancement 

Development 

NASA 





NACA (to 1958 






IS. PROBLEM CONSTRAINTS 

All other things being equal, 
rank, each of the following types of 
organization in order of their ability 
to focus on achieving solutions to 
specific real problems. Proceed down 
the column and write one number in each 
box, from 1 (most able) to 5 (least 
able) for each type of organization. 



Ability 
to Focus 

MAN COS 


SERV COS 


NON-PROFS 


UNI VS 


GOV LABS 




Basic 

Research 

Applied 

Research 

Technology 

Advancement 

development 

MAN COS 





SERV COS 





NON-PROFS 





UNIVS 





GOV UBS 
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16. DISSEMINATION OF RESEARCH AND TECHNOLOGY 

Rank the following types 
of organization in order of their 
willingness to provide outside 
organizations with FULL INFORMA- 
TION on government funded 
RESEARCH and TECHNOLOGY, ever, 
beyond the information contained 
in published reports required by 
the government. Proceed down 
the column and write one number 
in each box, from 1 {most 
willing) to 5 (least willing) 
for each segment of the aeronautical 
community. 

MAJOR EXPERIMENTAL FACILITIES 

A major experimental facility is defined here as a general 
experimental facility costing about a million dollars or more. Please 
indicate your opinion in regard to the following statements by circling 
an appropriate number following each statement. * 

17. "Major aeronautical experimental facilities built by the 
government should be managed by organizations that will provide usage 
and/or results to the entire aeronautical community." 


4 -3 -2 

-1 .0 1 

2 

3 4 

» i » 

strongly 



strongly 

disagree 

neutral 


agree 


18. "Organizations in the private sector can more effectively manage 
and control major government-built facilities than can the government." 

4 -3 -2 -1 0 1 23 4 

T » " t i i i i i r 

strongly strongly 

disagree neutral agree 

19. "'ihere should be resident research groups located at major 
aeronautical experimental facilities built by the Federal Government, if 
these facilities can be useful for basic or applied research." 

4 -3 -2 -1 0 1 2 3 4 

r ■ i 1 • \ * i \ i r 

strongly strongly 

disagree neutral agree 


. . t.. 

Avillingness to provide 
information 

MAN COS 


SERV COS 


NON- PROFS 


UNIVS 

■ ... 

GOV UBS 

. 





3 
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20. Roughly indicate the percentage of the capital value of all 
I’.ajor aeronautical experimental facilities built by the government 
that you think should be located at and under the managerial control 
of the following types of organizations. ...... 


Manufacturing Cos: % 

Service R&D Cos: % 

Non-Profit 

Institutions % 

Universities: % 

Government Labs: % 

Government 

Test Centers: % 

Total: 100% 


21. TECHNICAL GUIDANCE OF CONTRACTS 

In the following matrix, rank the various segments of the 
aeronautical R6D community in order of how well you believe they 
can provide the government with IMPARTIAL TECHNICAL GUIDANCE on 
CONTRACTS with other organizations for each of the indicated categories 
of work. Contract technical guidance includes: technical advocacy of 

needed new contracts, technical evaluation of proposals, selection of 
competent- contracts, technical guidance of contractors, checking 
technical work, helping solve technical problems and evaluating results. 
Proceed down each column and enter one number in each box. Rank ’each 
type of organization in order of ability to provide government with 
impartial technical contract guidance by entering a number from 
1 (most able) to 6 (least able) for each of the categories of work 
shown at the head of the column. See pages 3-5 for definitions of any 
terms that are unfamiliar. 

IMPARTIAL TECHNICAL GUIDANCE ON CONTRACTS 
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22. • DISTRIBUTION OF WORK 

This question is intended to reflect your view on roughly 
the extent to which different segments of the aeronautical RfiD 
community should participate in each category of federally funded 
work in order, to best advance American aeronautics. Assume that there 
is a set amount of federal funding available for each category of work. 
Write the percentage of these funds that you feel should be expended by 
each segment of the aeronautical R5D community. 

The introduction (Page 1) defines the funding break-down 
relative to government laboratories and their contractors. You might ■ • 
refer back to it if this is not clear now. It might also be a good idea 
at this point to review the definitions of the work categories on 
pages 3-4. 


DISTRIBUTION OF WORK IN EACH AREA 
(Percent- %) 



Basic 

Research 

Applied 

Research 

Technology 

Advancement 

Development 


(Sample) 

MAN COS 






10 

SERV COS 






25 

NON-PROFS 






IS 

UN1VS 






40 

COV LABS 






10 

TOTAL 

. 

100% 

100% 

100% 

100% 


100% 






23 . This question asks how NASA/NACA, from what you know of then, 
appears to have apportioned in-house expenditure for the various 
categories of aeronautical work. 

Proceed down each of the first two columns in the matrix below. 
Indicate the percentage of NASA/NACA expenditures for in-house 
aeronautical efforts that you. think probably related to each category 
of work. It alight be worthwhile to again review the definitions of 
work on Pages 3-4. 


RELATIVE EMPHASIS IN NASA/NACA CENTERS 



Apparent 

Distribution Expenditures 

Recommended 
Distribution 
Of NASA's work 

NACA (to 19S8) 

NASA 

Basic Research 




Applied Research 




Technology 


_ . . . _ 


Development 




TOTAL 

100% 

100% 

100% 


Next, go to the third column in the preceding matrix. For each 
category of work, write in the percent of NASA's in-house expenditures 
for aeronautics that you think would probably result in the greatest 
contribution to American aeronautics. 
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APPENDIX C-2 


AGENDA • * 

Thursday, 5 August 1976 
MORNING SESSIONS 

8; 00 - 8:25 INTRODUCTORY SESSION 

Location: Regency Room, Rickey's Hyatt House, see 

enclosed map. (.Coffee and rolls will be available 
in the room starting at 7:30 a.m.) 

Opening Comments: H. Harvey Album 5 minutes 

/ . Chairman 

Stewart Fliege 5 minutes 

Pepperdine University 

J. Lloyd Jones 20 minutes 

NASA Ames Research Center 

8:25 - 8:45 WORKSHOP METHODOLOGY AND DEFINITIONS OF TERMS 


• Definitions of organizational segments 

• Definitions of work categories 

• Procedures 

• Role of Observers 

The morning session is devoted to establishing a 
consensus of views on the following questions:. 

a. Which segments of the aeronautical R&D 
community should conduct federally supported 
work in (1) basic research, (2) applied 
research, (3) technology advancement, and 
(4) development? 

b. What are the commonly recognized and agreed 
upon reasons why such work should be 
conducted by particular segments of the 
aeronautical R&D community? 

8:50-9:50 HOMOGENEOUS GROUPS 

The. participants will be divided into five groups. Each 
group will consist of representatives of a single segment 
of the aeronautical R&D community. The groups will meet 
in separate rooms as follows: 

Manufacturing Companies Rm. A Universities Rm. D 

Service R&D Companies Rm. B Government Labs Rm. E 

Non-profit Institution Rm. C 
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- 10:20 REVIEW AND EXPANSION OF RATIONALE 


The groups will go through a rotation process by which 
they will review and expand upon each other's logic. 

Room assignments: 

Manufacturing Companies Rm. B Universities Rm. E 


Service R&D Companies 


Rm. C Government Labs Rm. A 


10:20 - 


Non-profit Institutions Rm. D 
10:35 BREAK 


10:35 - 11:00 Room Assignments: 



Manufacturing Companies 

Rm. C 

Universities 

Rm. 

A 

: 

Service R&D Companies 

Rm. D 

Government Labs 

Rm. 

B 

j- 

Non-profit Institutions 

Rm. E 




; ' 11:05 - 11:30 

Room Assignments: 





i ' 

K- 

Manufacturing Companies 

Rm. D 

Universities 

Rm. 

B 


Service R&D Companies 

Rm. E 

Government Labs 

Rm. 

C 

l '' 

J- 

Non-profit Institutions 

Rm. A 




!■ ■ 11:35- noon 

d 

Room Assignments: 

r ■ 




V 

Manufacturing Companies 

Rm. E 

Universities 

Rm. 

C 

s| ■ ■ . . 

Service R&D Companies 

Rm. A 

Government Labs 

Rm. 

D 


Non-profit Institutions 

Rm. B 





- 1:00 LUNCH 


AFTERNOON SESSIONS 


- 1:50 GENERAL SESSION . Regency Room 

o Review of morning results 
9 Explanation of objectives and methodology 
for afternoon 

The afternoon session is aimed at establishing a con- 
sensus of views, if possible, on the following questions: 

a. What is the relative importance of the agreed-upon 
reasons for various segments of the aeronautical 
R&D community conducting given categories of 
government funded work? 

b. Roughly, what should be the percentage distribution 
of federally supported work (federal funding), among 
the various segments of the aeronautical community 
for each category of work? 
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2:00 - 3:20 MIXED GROUPS 


The participants will again be divided into five groups; 
however, this time each group will consist of one repre- 
sentative from each of the five segments of the 
aeronautical R&D comnunity. Each of these groups wiil 
meet in a separate room where the participants will, 
concentrate on a particular category of work. However, 
one group will be concerned specifically with in-house 
work by NASA Laboratories. 


Room assignments: 

Basic Research Rm. A 
Applied Research Rm. B 
Technology Advancement Rm. C 


Development Rm. D 
NASA Rm. E 


3:20 - 3:35 BREAK 

3:35 - 5:00 GENERAL SESSION , Regency Room 

• Reports on results of mixed group sessions 


• Registering individual views (written) 

5:00 - 5:20 FEDERAL FUNDING FOR AERONAUTICS - GENERAL DISCUSSION 

o Review disposition of federal funding 
(U. R. Chirichiello) 

• Concluding remarks * 

(H. H. Album) 





\. 
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do not 
write 
in this] 
column 


APPENDIX C-3 

EXAMPLE OF FLIP CHART 

TECHNOLOGY ADVANCEMENT/UNIVS 


A. The training of students for the later practice of 
engineering is furthered by university activity in 
technology advancement. 

B. Experience has shown in some cases, that the Government 
can obtain outstanding technology advancement from 
universities. 

C. The effort in universities to achieve large jumps in 
technology can serendipitously stimulate the development 
of fundamental fields in engineering science. 

D. The presence in a university of fundamental scientists 
contributes to the innovative thinking process essential 
~to the achievement of large technology jumps. 











RATIONALE 

FOR 

CONDUCTING GOVERNMENT SUPPORTED AERONAUTICAL WORK 


WORK CATEGORY: 


TECHNOLOGY ADVANCEMENT 


BY SEGMENT: 


REASONS FOR GOVERNMENT SUPPORT OF WORK BY THIS SEGMENT 


The training of students for the later practice of engineer- 
A. ing is furthered by university activity in technology 
advancement. 


Experience has shown in some cases, that the Government can 
obtain outstanding technology advancement from, universities. 


The effort in universities to achieve large jumps in tech- 
C. nology can serendipitously stimulate the development of 

fundamental fields in engineering science. 


Serv.Co. Non-Prof. Univ'sl Gov. Labs 


MS 

U> 


EXAMPLE OF HAND-HELD RATIONALE SI 
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APPENDIX C-5 

RANKING METHODS AND COMPUTATIONS 

INDIVIDUAL RANKINGS ; 

Tne aeronautical leaders ranked many factors in the survey 
questionnaire and during the workshop. For instance, in the question- 
:■ naire, the participants ranked various types of organizations on the 
relative quality of R&D they tend to perform. Another important exam- 
ple was at the conclusion of the workshop when the participants ranked 
the basic rationale for government use of particular sectors of the 
aeronautical R&D conmunity for conducting given categories of work, In 
all cases, the number 1 was used to indicate the highest rank in terms 
of merit and a larger number indicated a lower degree of merit. The 
numerical rankings extended from 1 to 5 or 6. The participants were ^ 
not to skip numbers or use tied numerical rankings. 

The ranking procedures for the survey questionnaire were the 
same as those for the workshop, with only two exceptions. The partici- 
pants could use two additional types of rankings during the workshop. 
After ranking five statements of rationale from 1 to 5, they could then 
use blanks to indicate that rationale statements were valid, but of 
less significance than the numerically ranked items. They could also 
use a blackball (a) to indicate that statements of rationale were 
incorrect or invalid. 

i . ; 

There were occasional ties, omissions, and other errors in the 
rankings submitted by the participants. Ties were incorrectly awarded 
for 0.8 percent of the total number of survey questionnaire ranks 
returned by all participants and in less than 0.1 percent of the total 
individual workshop ranks. Ranks were inappropriately left blank in 
5.7 percent of the total number of rankings returned with the survey 
questionnaires. The ties and omissions could have reflected the parti- 
cipants' inability to differentiate among rankings for certain vari- 
ables, a failure to understand the instructions, or simply a lack of 
interest. Ties and omissions were corrected in cases where the 
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intended relative rankings were obvious. The remainder of the errors 
and omissions were taken into account in the scoring procedures. This 
procedure essentially treated ties and omissions as if the participants 
could not distinguish among the relative rankings of the associated 
variables. . . - 


SCORING INDIVIDUAL RANKINGS 


Individual rankings were scored in a manner intended to provide 
complete and consistent sets of rankings for determining the statisti- 
cal significance of the individual rankings and to formulate group con- 
sensus rankings. The scoring was done as follows: 

Step 1. Whenever it was possible, errors were corrected 
to reflect the exact ranking order intended by 
the participant. 

Step 2. Next, ties were given scores equal to the average 
of the ranks they would have had if they had been 
ranked in order, beginning after the next lowest 
discrete numerical rank. 

Step 3. The scores of the other discrete numerical rankings 
were then adjusted upward or downward as would have 
been necessary if the ties had been ranked. 

Step 4. Blanks were then given the next greatest scores. 

The blanks were scored as the average of the ranks 
they would have had if they had been ranked in 
order beginning with the next highest rank possible 
after all the ties had been ranked in order. 

Step 5. Blackballs (.a) were scored last. They received 
the highest numerical score possible for a given 
ranking by an individual. Blackballs were set 
equal to the average of the numerical ranks they 
would have had if they had been ranked in order tip 
to the greatest possible value for that ranking. 

The three examples on the following page illustrate this- 
scoring procedure. 



OMGWAL PAGE* 

OF POOR 








VARIABLES 


SCORING STEPS 

A 

B 

C 

D 

E 


Participant's ranking 

2 

5 

4 

3 

5 


1. Corrections 

1 

5. 

3 

2 

5 


2. Scoring ties 

1 

4.5 

3 

2 

4.5 


Example 2: - 



VARIABLES 



SCORING STEPS 

A 

B 

C 

D 

E 

F 

Participant's ranking 

2 

3 

3 

4 

5 

4 

1. Corrections 

1 

3 

3 

4 

5 

4 

2. Scoring ties 

1 

2.5 

2.5 

4.5 

5 

4.5 

3. Adjusting scores 

1 

2.5 

2.5 

4.5 

6 

4.5 



SCORING STEPS _A_ _B C D E F G H I_ 

Participant's ranking 251 3 • 3 • 

1. Corrections 241 3 e 3 e 

2. Scoring ties 24 3.5 « 3.5 e 

3. Adjusting scores 251 3.5 e 3.5 a 

4. Scoring blanks 2 5 1 6.5 3.5 s 6.5 3.5 o 

5. Scoring blackballs 2 5 1 6.5 3.5 8.5 6.5 3.5 8.5 

GROUP CONSENSUS RANKING 

A group consensus ranking is a measure of the opinion of a 
group, as a whole,, in regard to the correct ranking of items that are 
ranked in order by the individual members of the group. The group 
consensus ranking is the best estimate of group opinion when the rela- 
tive rankings by members of a group are statistically significant 
(71:100 to 102). In the current study, this is taken as a level of 
probability of 95 percent or higher that the differences in ranking, 
among all the i tens were not due to chance. 





The group consensus ranking is operationally defined by the 
following measurements and procedures for calculating them: 

Step 1. The scores discussed in the previous section were 
truncated to reduce distortions caused by high 
values assigned to blanks and blackballs. This 
often occurred when the number of judges was 
smaller than the number of items bein? ranked. 

Blanks were limited to a maximum value of 6 and 
blackballs were assigned a maximum value of 7. 

Step 2. Then a group's scores for any one item were averaged. 
This was done individually for each item that was 
being ranked by the group. 

Step 3. If four-fifths of a group awarded any item a black- 
ball then that item received a final group-consensus 
rank of a (meaning invalid or incorrect). 

Step 4. The other items being ranked were then awarded 

positive integer ranks in the same numerical order 
as the average scores. This began with the 
number 1 and continued progressively toward higher 
numbers. . ' a. . 

Step 5. Equal tied ranks were awarded in cases where the 

averages fell within a specific maximum interval of 
each other. This interval was taken as 0.25 based 
on a qualitative comparison of the data. The lowest 
average score involved established the base of this 
interval . ■ 

Step 6. Ties were given the minimum numerical value they 
would have had if they had been ranked in order. 

Step 7. The remaining rankings were then adjusted to the 

values they would have had if the ties had all been 
ranked in order. This process skips the other 
intermediate ranks which would have been spanned by 
the ties. 


ORIGINAL PAGE IS 

OF POOR QUALITYi 
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The following example illustrates the process for calculating 
group consensus rankings for the workshop results. 

Example for a Single Judge ITEMS 


A B C D E F G H I 


Score 

2 

5 

1 

6.5 

3.5 

8.5 

6.5 

3.5 

8.5 

Truncating 

2 

5 

1 

6 

3.5 

7 

6 

3.5 

7 

Example for a Group of 










Judges 




ITEMS 






A 

B 

C 

D 

E 

F 

G 

H 

I 

#1 

2 

5 

1 

6 

3.5 

7 

6 

3.5 

7 

#2 

7 

6 

2 

5 

3 

7 

7 

4 

1 

#3 

1 

5 

2 

6 

4 

7 

7 

3 

6 

#4 

1 

4 

2 

5 

3 

6 

7 

6 

6 

#5 

2 , 

6 

1 

5 

3.5 

6 

7 

3.5 

7 

Averaging 

. 2.6 

5.2 

1.6 

5.4 

3.4 

6.5 

0 

4 

5.4 

Rankings 

2 

5 

1 

5 

3 

7 

0 

4 

5 

Adjusting for Ties 

2 

5 

1 

5 

3 

8 

0 

4 

5 
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APPENDIX C-6 

EVALUATION AND ASSESSMENT FORMS 



I 

a 




RELATIVE EMPHASIS IN NASA/NACA CENTERS 



DISTRIBUTION OF WORK IN EACH AREA 
(Percent %) 










INDIVIDUAL ASSESSMENTS 


DO NOT INDICATE YOUR NAME 


Check below the type of organization that best describes the principal 
nature of your company, division, institution or government organization. 

(Check one) 


r i Manufacturing Company 
f~j Service R S D Company 
H - Non-Profit R 5 D Inst. 


r i Government Lab 

FI Government R 6 D Fund. Org. 

FI Other 

(Write in) 


: • ' : '/ 

Check below the kind of aeronautical work that you are primarily concerned 
about in your current position. 


; Basic Research 

i I Applied Research 

\~! Technology Advancement 


H Product Development 

i 

ri Other 

(Write in) 









APPENDIX C-7 



PARTICIPANTS' REACTION TO TRIBUTARY TECHNIQUE 



RESPONSES TO : "WERE COMMUNICATIONS IMPROVED BY THE 

WORKSHOP METHODOLOGY RELATIVE TO THE METHODOLOGY OF 
PREVIOUS ADVISORY GROUPS YOU HAVE PARTICIPATED IN, ON 
HIGHLY SUBJECTIVE TOPICS, WHERE ADVISORS HAD NATURALLY 
DIVERSE GROUP INTERESTS?" 









FIGURE IS: CRITIQUE OF TIME FACTOR 
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B£SPQHS £S T O: "WHAT IS TOUR OPINION AT THIS POINT OF 

THE POTENTIAL VALUE Or THE RESULTS OF THE WORKSHOP YOU 
HAVE JUST COMPLETED?** 



PROBABLY DO NOT MUCH OF IT PROBABLY 

LOW OP. KNOW PROBABLY . VALUABLE 

MARGINAL' - VALUABLE 

VALUE 


FIGURE 19: EXTENT OF OPT IMISM ON USAGE OF RESULTS; 
END OF WORKSHOP 







